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ABSTRACT 

Smart  Systems  for  Logistics  Command  and  Control  (SSLC2)  is  an  Air  Force 
Research  Laboratory  Warfighter  Readiness  Research  Division  program  to  develop  and 
apply  technologies  to  collect  the  critical  information  required  to  effectively  manage 
logistics  resources  in  support  of  combat  operations.  The  purpose  of  SSLC2  is  to  develop 
capability  requirements  that  employ  technologies  and  techniques  to  autonomously  collect 
and  fuse  critical  data  in  order  to  create  decision  quality  information  and  effectively 
present  information  to  support  cognitive  tasks  performed  by  logistics  and  operations 
decision-makers. 

SSLC2  provides  support  to  Expeditors  and  Production  Supervisors  (ProSupers)  on  the 
fiightline.  Spiral  Three  research  focused  on  the  fusion  of  information  and  presentation  of 
the  information  through  interactive  visualizations  to  support  situation  awareness,  decision 
making  and  collaboration  when  Expeditors  and  ProSupers  are  faced  with  multiple 
complex  tasks  and  decisions.  A  user-centered  design  approach  was  used  to  support  the 
development  of  visualizations.  A  cluster  analysis  technique  was  incorporated  to 
determine  how  to  integrate  large  amounts  of  data  for  arrangement  in  the  interactive 
visualizations.  A  simulation  test  bed  was  created  to  allow  Expeditors  and  ProSupers  to 
evaluate  SSEC2  concepts  and  provide  their  opinions  on  the  effectiveness  of  information 
integration,  visualizations,  support  for  situation  awareness,  decision  making, 
collaboration,  resource  visibility  and  resource  allocation.  A  field  demonstration  was 
conducted  at  the  Toledo  Air  National  Guard  which  included  1)  an  interactive  simulation 
that  integrated  fiightline  data  with  smart  sensors  that  provided  resource  location  and 
status  and  2)  a  demonstration  of  live  tags  on  the  fiightline.  The  simulated  study  collected 
user  opinions  via  a  questionnaire  and  comfort  ratings,  and  timing  results  for  situation 
awareness  probe  questions.  The  study  results  indicated  that  overall  subjects  found  the 
SSEC2  system  to  be  effective  and  useful.  Many  issues  reported  by  subjects  were  related 
to  SSEC2  requirements  that  have  not  been  demonstrated  but  were  already  considered 
within  the  design.  This  report  summarizes  recommendations  related  to  smart  sensors, 
user  interface,  portability,  trend  analysis,  fiightline  performance  metrics,  collaboration, 
configuration  tracking,  automation,  integration  with  existing  systems,  resource  conflict 
checking,  and  safety  driven  tasks.  Euture  research  needs  and  research  contributions  are 


VI 


also  described.  The  SSLC2  program  demonstrated  that  commercial  off-the-shelf  (COTS) 
applications  designed  without  thought  to  integration  with  existing  systems  where 
information  may  not  be  ideal  can  potentially  cause  problems  or  even  rejection  of  the 
sensing  technology.  Integration  of  information  is  a  key  to  the  successful  use  of  sensing 
technologies. 
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1  Summary 

Smart  Systems  for  Logistics  Command  and  Control  (SSLC2)  is  an  Air  Force 
Research  Laboratory  Logistics  Readiness  Branch  (AFRL/RHAL)  Warfighter  Readiness 
Research  Division  program  to  research,  develop  and  apply  technologies  to  collect 
the  critical  information  required  to  effectively  manage  logistics  resources  in  support 
of  combat  operations. 

SSLC2  developed  recommendations  and  requirements  to  transition  concepts  to 
existing  Air  Force  systems.  The  research  focused  on  ways  to  support  logistic  decision 
makers  in  complex  fiightline  operations  through  the  fusion  of  existing  fiightline 
information  with  smart  sensor  information.  Specifically,  Radio  Frequency  Identification 
/  Real  Time  Location  Systems  (RFID/RTLS),  information  was  presented  through  the  use 
of  interactive  visualizations.  The  SSLC2  program  was  divided  into  three  spirals.  This 
report  focuses  on  the  research  conducted  during  Spiral  Three.  The  Spiral  Three  team 
consisted  of  MacAulay  Brown,  HumanWise,  Inc.,  and  Alion  Science  and  Technology, 
Inc. 


The  goal  of  SSLC2  Spiral  3  was  to  investigate  ways  to  support  logistic  decision 
makers  in  complex  fiightline  operations  through  the  fusion  of  existing  fiightline 
information  with  smart  sensor  information.  The  focus  was  on  providing  support  to 
Expeditors  and  Production  Supervisors  (ProSupers)  on  the  fiightline.  This  research 
focused  on  the  fusion  of  information  and  presentation  of  the  information  through 
interactive  visualizations  to  support  situation  awareness,  decision  making  and 
collaboration.  The  following  five  research  objectives  were  met: 


Objective  1: 
Objective  2: 

Objective  3: 
Objective  4: 

Objective  5: 


Develop  smart  sensors  for  logistics  operations. 

Develop  a  method  for  determining  how  data  should  be  fused  for 
presentation  that  focuses  on  supporting  decisions  and  collaboration. 

Obtain  user  opinions  of  deployed  sensors  in  the  field. 

Develop  a  simulation  test  bed  and  evaluate  user  feedback  on  integrated 
data  fusion  decision  support  concepts. 

Develop  recommendations  for  transition  to  existing  systems. 
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The  program  started  with  information  gathered  from  previous  spirals  whieh  provided 
cognitive  work  models  of  the  Expeditor  and  ProSuper  and  other  system  and  user 
requirements.  A  user-centered  approach  was  used  to  develop  interactive  visualizations 
which  were  programmed  into  a  simulation  test  bed  that  fused  information  from  RFID 
tags  as  well  as  simulated  smart  sensors  with  flightline  information  [5]  [7].  A  method  was 
created  to  support  how  data  should  be  fused  for  presentation.  A  rating  scale  was 
developed  to  allow  Subject  Matter  Experts  (SMEs)  to  rate  the  importance  of  specific  data 
to  decisions.  A  cluster  analysis  was  performed  on  the  ratings.  The  information  from  the 
cognitive  work  analysis  (CWA)  and  results  of  the  cluster  analysis  were  used  to  design 
interactive  visualization  concepts.  The  concepts  were  then  presented  to  SMEs  who 
provided  feedback  and  indicated  that  the  design  concepts  appeared  useful  and  effective. 
Based  upon  the  design  concepts  a  detailed  design  was  developed  into  a  simulation  test 
bed.  Elpon  completion  of  the  test  bed,  a  study  was  conducted  to  obtain  user  opinions 
using  the  simulation  test  bed  at  the  180**'  FW  Toledo  Air  National  Guard  at  Toledo 
Express  Airport,  Swanton,  OH  in  Swanton,  Ohio  (to  be  hereby  referred  to  as  Toledo 
ANG).  A  live  sensor  tag  demonstration  occurred  concurrently. 

The  simulation  included  information  related  to  17  aircraft  with  sorties  and 
maintenance  for  a  one  week  (5  day)  period.  Five  subjects  interacted  with  the  system 
using  a  tablet  PC  with  a  mouse  and  keyboard  to  solve  problems,  collaborate  with  other 
personnel,  assign  resources,  make  maintenance  fix/swap  decisions,  and  oversee 
scheduled  and  unscheduled  maintenance.  The  study  took  approximately  one  and  one-half 
hours  to  complete  per  subject.  Subjects  filled  out  a  questionnaire  to  provide  their 
feedback.  During  the  simulation  subjects  were  asked  situational  awareness  questions 
related  to  information  on  the  flightline  and  the  time  it  took  to  answer  was  recorded. 
Subjects  were  also  asked  to  indicate  how  comfortable  they  felt  with  the  system  as  time 
progressed. 

The  feedback  received  from  subjects  and  others  who  viewed  the  live  demonstration 
was  very  positive.  The  questionnaire  results  indicated  that  subjects  considered  SSEC2 
concepts  and  fusion  of  information  from  sensors  with  fiightline  data  to  be  effective  and 
useful  and  thought  SSEC2  would  help  support  situation  awareness,  decision  making,  and 
collaboration.  Efsers  indicated  that  this  system  would  be  ideal  for  current  deployed 
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operations  where  they  have  been  having  diffieulty  finding  all  of  their  resources.  The  lead 
ProSuper  at  Toledo  ANG  indicated  that  it  takes  15  minutes  to  collect  the  information 
needed  to  make  a  decision  about  whether  or  not  an  aircraft  will  meet  a  sortie.  With  this 
system  he  stated  that  they  could  make  that  decision  in  seconds.  He  also  stated  the  system 
could  reduce  the  time  to  make  decisions  and  assign  resources  to  maintenance  tasks  by  up 
to  60%. 

This  program  successfully  developed  recommendations  for  future  spirals  as  well  as 
requirements  for  transition  to  current  Air  Force  systems.  The  implementation  of  this 
technology  would  help  the  Air  Force  towards  its  goal  of  becoming  more  agile. 


1.1  RESEARCH  PUBLICATION  ACCOMPLISHMENTS 

Throughout  the  SSLC2  research  project  we  have  published  our  findings  and 
presented  papers  and  posters  to  scientific  peers.  The  papers  and  posters  are  listed  below. 


•  Gallimore,  J.J.,  Maki  A.,  Faas,  P.,  Seyba,  J.,  Quill,  L.,  and  Matthews,  E.  (2005).  The 
Need  For  A  Human-in-the  Loop  Simulation  Testbed  for  Logistics  Decision  Support 
Research.  \a  Proceedings  of  the  Summer  Simulation  Conference,  84-89. 

•  “How  Can  We  Make  Logistics  Command  &  Control  Technologies  Smart”;  Integrated 
System  Health  Management  Conference  Poster;  9-11  August  2005. 

•  “Smart  Systems  for  Logistics  Command  and  Controf’;  National  Annual  Society  for 
Logistics  Engineers  Conference;  16-18  August  2005. 

•  “Virtual  Space  Logistics  Readiness  Center”;  National  Annual  Society  for  Logistics 
Engineers  Conference;  16-18  August  2005. 

•  Laas,  P.,  Seyba,  J.,  Young,  L,  Gallimore,  J.J.,  Quill,  L.,  Matthews,  E.,  and  Cagle,  R. 
(2006).  Collaborative  Logistics  on  the  Military  Llightline.  International  Symposium 
on  Collaborative  Technologies  and  Systems  (CTS'06),  215-219. 

•  Gallimore,  J.J.,  Quill,  L.,  Cagle,  R.,  Gruenke,  J.,  Hosman,  C.,  Matthews,  E.,  Laas,  P., 
Seyba,  J.  and  Young,  I.  (2006).  User  Leedback  on  REID  and  Integrated  Llightline 
Data  for  Maintenance  Decisions.  In  Proceedings  of  the  Institute  of  Industrial 
Engineers  Annual  Conference.  May,  Orlando.  CDROM. 

•  “Smart  Systems  for  Logistics  Command  and  Control”,  Integrated  System  Health 
Management  Conference  Poster;  14-17  August  2006. 
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•  Gallimore,  JJ.  Matthews,  E.M.,  Cagle,  R.  Faas,  P.,  Seyba,  J.  and  Whited,  V.  (In 
Press).  Integrating  Sensor  Data  with  System  Information  Via  Interactive 
Visualizations.  In  Proceedings  of  the  2007  International  Human  Computer 
Interaction  Conference,  July  23-28,  2007,  Beijing,  China. 

•  GRACAR  Corporation  (2006).  Smart  Systems  for  Logistics  Command  and  Control 
(SSLC2):  Interim  Final  Report,  Spiral  1.  Contract  Number  FA8650-04-C-6404. 
Dayton,  OH. 

•  GRACAR  Corporation  (2006).  Smart  Systems  for  Logistics  Command  and  Control 
(SSLC2):  Interim  Final  Report,  Spiral  2.  Contract  Number  FA8650-04-C-6404. 
Dayton,  OH. 

2  Introduction 

Logistics  support  for  the  U.S.  Military  is  a  complex,  time  dependent,  yet  critical  task. 
Creating  Agile  Combat  Support  (ACS)  for  the  warfighter  requires  real  time  integrated 
information  systems  to  support  human  decision  making.  Through  the  years,  support  tools 
have  been  created  to  support  logistics  problems.  Many  of  the  systems  created  have  been 
standalone  systems  that  do  not  share  data.  In  the  late  1960’s  the  Logistics  Composite 
Model  (LCOM)  was  created  to  allow  Air  Force  leadership  to  evaluate  new  weapon 
systems  or  modifications  impacts  on  logistic  resources  and  model  for  logistics  manpower 
requirements  [1].  Integrated  Maintenance  Information  Systems  (IMIS)  was  developed  in 
the  1990’s  to  integrate  maintenance  information  such  as  technical  manuals,  diagnostic 
instructions,  work  orders,  supply  availability,  etc  in  a  single  hand-held  computer  based 
integrated  system  [22].  In  1999,  AFRL  initiated  the  LOCIS  program  to  research  ways  of 
utilizing  existing  data  from  current  systems  such  as  Core  Automated  Maintenance 
System  (CAMS)  and  IMIS,  and  presenting  the  information  to  the  Operations  Group  (OG) 
Commander,  Logistics  Group  (LG)  Commander,  and  their  respective  staff  The  focus  of 
the  LOCIS  program  was  on  information  fusion,  decision  support,  and  dynamic  user 
interfaces  [13,  14].  This  successful  program  was  a  first  step  in  developing  integrated 
information  systems  to  support  human  decision  making. 

The  vision  is  to  develop  SSLC2  to  fuse  large  amounts  of  location  and  status 
information,  provide  situation  assessment,  and  monitor  equipment  and  human 
performance  in  such  a  way  that  human  operators  and  technology  work  together  as  a 
symbiotic  team.  This  vision  is  a  warfighter  requirement  -  the  right  information  delivered 
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at  the  right  time  at  the  right  level.  Advanees  in  smart  sensors,  computing  technology, 
information  networks,  and  research  in  human  cognition  and  performance  will  help  us 
move  in  this  direction.  Real-  time  sensing  technologies  can  provide  myriads  of 
information  to  help  support  Air  Force  logistics.  For  example,  RFID  technology  is  being 
used  to  improve  In-  Transit  Visibility  (ITV)  of  equipment  and  supplies  during 
deployment  [21].  RFID  tagging  is  becoming  standard  within  many  commercial  supply 
chains.  The  Autonomic  Logistics  System  (ALS)  is  a  concept  to  allow  automatic  detection 
of  aircraft  system  faults  or  system  deterioration  to  provide  active  logistics  maintenance 
information  versus  the  current  reactive  approach  [18].  War  fighter  physiologic  and 
cognitive  monitoring  is  being  utilized  to  develop  smart  systems  to  adapt  to  the  war 
fighters  needs  [15,  20]. 

AFRL  is  interested  in  incorporating  real  time  sensing  technologies  to  improve 
fiightline  logistics  support.  This  technology,  as  well  as  others,  has  the  potential  to 
provide  significant  impact  to  fiightline  decisions.  Although  off-the-shelf  RFID/RTLS 
technology  provides  immediate  location  information  on  tagged  resources,  and  allows  a 
decision  maker  to  easily  find  these  resources;  in  its  stand  alone  form  it  is  merely  another 
information  stovepipe.  So  while  near  real  time  information  is  available,  it  will  not 
necessarily  improve  fiightline  decision  making.  The  challenge  with  sensor  technologies 
is  to  provide  the  correct  information,  at  the  right  time,  in  the  right  format  to  improve 
human  decisions  and  system  performance.  Too  often,  adding  the  technology  without 
considering  the  impact  creates  “shelf-ware.”  Often  development  of  new  technologies, 
such  as  RFID,  occurs  so  rapidly  the  technology  push  can  lead  to  flawed  technology¬ 
centric  systems  lacking  a  human  centered  engineering  approach.  In  the  end  humans  will 
be  the  ones  using  the  technologies  and  systems  to  perform  important  tasks,  make  critical 
decisions  and  fulfill  mission  requirements.  Therefore,  human  performance  must  be 
central  to  system  development. 

With  these  considerations  in  mind  the  AFRL  initiated  the  SSLC2  Advanced 
Technology  Demonstration  Program  researching  technologies  and  techniques,  facilitated 
through  software,  to  craft  information  enabling  the  warfighter  to  become  more  agile, 
productive  and  smart  about  critical  logistic  resources  required  for  combat  operations.  It 
has  been  recognized  that  decision  support  tools  must  be  created  to  fully  exploit  real  time 
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information.  The  SSLC2  program  focuses  on  human  performance  in  complex,  time 
constrained,  high-stress,  decision  making  situations  and  on  leveraging  technology  to 
assist  the  human  decision-maker  by  capturing  and  displaying  data  in  innovative  ways  to 
promote  more  effective  decision  making  on  the  flightline.  The  overall  goal  is  to  provide 
decision  support  on  the  flightline  to  improve  sortie  production  resource  allocation, 
personnel  and  resource  scheduling,  and  decision  making  at  multiple  levels. 

2.1  THE  PROBLEM 

Logistics  and  sortie  production  mission  success  depend  on  resources  (people, 
equipment,  and  information).  Yet  few  bases  manage  the  availability  and  performance  of 
their  resources  from  a  mission  perspective.  Resource  management  is  often  driven  by 
priorities  that  are  disconnected  from  the  primary  mission  objectives  and  end-user 
requirements.  Aerospace  Ground  Equipment  (AGE),  test  equipment  and  people  are 
employed  in  stovepipe  fashion,  leading  to  fragmentation  and  “silos”  of  infrastructure. 
This  situation  leaves  managers  with  no  visibility  into  overall  service  levels,  no  way  to 
proactively  detect  and  prevent  availability  and  performance  problems,  and  no  means  of 
quantifying  the  impact  of  downtime  on  the  pilots,  maintainers  and  the  overall  mission.  It 
adds  up  to  delays,  complexity,  over  allocation  and  end-user  frustration. 

The  problem  is  while  there  are  intelligent,  qualified  people,  equipment  information 
including  what  resource  is  needed,  the  resource’s  related  location,  or  its  status  is  not 
always  known.  Equipment  is  used  and  returned  to  the  ready  line  or  the  tool  crib  without 
any  indication  of  a  change  in  status.  Sometimes  equipment  is  used  and  then  left  in  place 
for  future  known  or  unknown  actions.  The  AGE  driver  is  left  to  wonder  if  this  equipment 
is  still  needed,  ready  for  return  to  the  ready  line,  or  in  need  of  servicing  or  repair.  People 
of  the  same  Air  Eorce  Specialty  Code  (AESC)  develop  additional  qualifications  or  higher 
levels  of  competency  in  various  tasks,  but  that  information  is  either  documented  in  a 
manual  record,  somewhere  in  a  wide  variety  of  databases  or,  in  the  case  of  levels  of 
competency,  not  documented  at  all.  Many  times  decisions  are  made  to  optimize  resource 
utilization,  not  sortie  production. 

To  meet  the  demand  for  agile  combat  support  a  variety  of  information  technology 
tools  are  being  developed  and  purchased  by  the  U.S.  Air  Eorce.  Many  of  these  off-the- 
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shelf  technologies  are  given  to  users  and  they  are  told  to  figure  out  how  to  use  them.  In 
many  cases  maintenance  personnel  are  developing  their  own  in-house  programs  to  deal 
with  the  information  they  use.  With  the  large  amounts  of  information  and  complex 
decisions  that  occur  in  the  air  base  environment,  it  becomes  crucial  to  involve  users  in  the 
development  of  technologies,  especially  decision  support  tools,  built  for  their  use. 

RFID  and  RTFS  are  examples  of  off-the-shelf  technologies  with  potential  to  improve 
the  maintenance  process.  RFID  is  sometimes  called  Dedicated  Short  Range 
Communication  (DSRC).  An  RFID  system,  at  a  minimum,  consists  of  three  components: 
an  antenna  and  transceiver  (often  combined  into  one  component  called  a  reader)  and  the 
tag,  of  which  there  are  two  types  (passive  and  active).  In  a  simple  RFID  system  a  passive 
tag  is  used  along  with  a  reader  which  is  usually  a  permanent  gateway  into  a  facility.  This 
will  only  allow  for  entry  and  departure  readings  from  a  defined  facility  or  area.  RTFS 
uses  RFID  technology  to  transmit  the  physical  location  of  RFID  tagged  objects.  RTFS 
systems  require  active  RFID  tags  to  be  attached  to  each  object  needing  tracked  and  Radio 
Frequency  (RF)  transmitters/receivers  located  throughout  the  facility  determine  the 
location  and  send  information  to  a  computerized  tracking  system.  RFID/RTFS 
technologies  with  active  tagging  have  the  potential  to  improve  awareness  related  to 
resource  location  by  showing  where  things  are.  If  the  required  piece  of  equipment  is 
sitting  on  the  flightline,  it  is  not  possible  to  tell,  within  the  current  applications,  what  it  is 
doing  out  there.  It  might  be  in  use,  out  of  fuel,  awaiting  a  high  priority  task  to  begin,  or 
ready  for  re-assignment.  These  capabilities  can  be  accomplished  through  customization 
of  the  off  the  shelf  RFID/RTFS  products. 

These  challenges  reflect  up  to  the  Major  Command  (MAJCOM)  and/or  Air  and  Space 
Operations  Center  (AOC)  in  the  broader  context  of  “will  that  base  meet  its  sortie 
schedule  and  support  the  mission?”  and,  “if  not,  what  is  the  impact?”  The  characteristics 
of  the  current  environment  include  little  cross-echelon  situational  awareness  due  to  a 
largely  manual  data  capture  and  decision  analysis  processes.  There  remains  very  little 
insight  to  wing-level  sortie  production  capability.  Decision  making  is  still  limited  by  the 
ability  to  assimilate  data  into  reasoned,  actionable  information.  In  the  end,  off-the-shelf 
Automatic  Information  Technologies  (AIT)  only  solves  a  small  part  of  the  situational 
awareness  and  decision  making  challenge. 
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Flightline  maintenance  for  sortie  production  can  be  considered  a  very  complex 
logistics  system.  The  ultimate  goal  is  to  improve  Air  Force  systems  to  create  a  more  agile 
Air  Force.  Complex  systems  require  the  management  of  resources  to  meet  multiple 
objectives  and  require  substantial  coordination  among  entities  within  and  outside  their 
own  subsystems.  Enhanced  sensor  technologies  have  the  potential  to  provide  additional 
information  to  support  decisions  related  to  the  management  of  logistics  support. 
Flightline  maintenance  requires: 

•  the  need  to  monitor,  plan  and  schedule; 

•  the  need  for  collaboration; 

•  the  need  to  make  real  time  decisions  and  changes  to  current  plans  and 
schedules; 

•  the  need  to  coordinate  among  many  groups  or  other  subsystems; 

•  the  need  to  predict  how  changes  may  affect  subsystems  and  schedules; 

•  the  need  to  coordinate  use  of  resources  to  meet  multiple  objectives; 

•  managing  large  amounts  of  information  needed  during  the  decision  process 
and 

•  the  sharing  of  similar  types  of  data. 

These  requirements  are  not  unique  to  flightline  maintenance,  but  are  issues  with  most 
existing  complex  systems  such  as  logistics  for  air  and  space,  health  care,  disaster 
management,  homeland  security,  to  name  a  few.  Therefore,  research  on  this  project  will 
be  of  beneht  to  other  complex  systems. 

2.2  SSLC2  PROGRAM  VISION 

The  vision  of  SSLC2  is  to  provide  personnel  decision  quality  information  from 
multiple  data  streams  including  real  time  location  and  status  data  for  monitoring 
capabilities,  as  well  as  algorithms  to  equate  resource  availability  to  operational  capability. 
Decision-makers  will  be  provided  insight  to  the  operational  impact  of  their  decisions.  The 
SSLC2  effort  is  researching  the  fusion  of  technology  with  information  to  provide 
decision  support  and  situational  awareness  aimed  at  demonstrating  the  capability  for 
improved  decision  making  that  can  affect  sortie  generation.  Functional  systems  that 
support  maintenance  data  collection  along  with  developing  systems  that  will  contain 
electronic  Technical  Orders  (TOs),  Enhanced  Maintenance  Operations  Center  (EMOC) 
support  systems,  or  even  direct  maintenance  support  systems  would  contain  SSEC2 
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capabilities.  In  this  way,  predictive  logic  can  be  applied  to  determine  the  appropriate 
resources  to  use  and  maintain  the  highest  possible  Mission  Capable  (MC)  rates.  The 
SSLC2  vision  incorporates  Interface  Requirements  Agreements  (IRA)  with  any  system 
that  collects  data  that  would  be  useful  in  the  management  of  people,  equipment,  and 
information.  For  example,  if  medical  appointments  are  contained  in  a  medical  system,  an 
IRA  can  be  drafted  allowing  the  extraction  of  data  without  violating  the  privacy  of  the 
medical  system. 

Another  vision  is  that  a  system  would  contain  job  guides,  the  TO  equivalent  of  a 
“How  To”  guide  for  major  maintenance  tasks  on  each  weapon  system.  These  job  guides 
detail  which  major  pieces  of  equipment  are  required  and  what  follow-on  maintenance  is 
necessary.  In  this  way,  maintenance  management  functions  can  determine  logical 
priority  for  task  accomplishment  with  a  maximized  MC  rate  for  the  end  product. 

SSLC2  should  continuously  monitor  all  assets  in  its  domain  and  report  a  variety  of 
status  indicators.  Some  of  those  indicators  may  be  current  operational  condition  (is  it 
running),  fuel  load  remaining,  built  in  test  status  or  perhaps  data  entered  by  the  last  user. 
This  means  maintenance  managers  will  not  only  know  what  is  needed  for  the  job  but  they 
will,  in  one  look,  know  where  it  is  and  if  it  is  serviced,  repaired  and  available;  in  short  “it 
is  the  right  resource  for  the  job.”  If  the  resource  is  fully  operational  and  ready  for  reuse, 
the  technician  has  only  to  put  it  back  where  it  belongs  or  have  the  AGE  driver  pick  it  up. 
SSLC2  has  already  assessed  and  published  its  status. 

Information  about  personnel  resources,  such  as  standard  AFSC-related  details,  would 
be  fused  from  the  appropriate  system  and  all  AF  Form  623  and  Job  Qualification 
Standard  (JQS)  data  should  be  available.  In  addition,  maintenance  managers  can  make 
personal  notes  about  workers  to  aid  in  selecting  the  correct  person. 

While  the  vision  is  to  provide  recommendations,  the  actual  decisions  and  selections 
would  be  firmly  in  the  hands  of  the  maintenance  manager.  The  goal  of  SSFC2  is  to 
provide  recommendations  that  optimize  sortie  production  even  if  it  appears  that  resources 
are  sub-optimized. 

Under  the  SSFC2  concept,  not  only  will  maintainers  know  where  things  are,  they  are 
provided  recommendations  as  to  which  resource  is  most  appropriate  for  the  task  at  hand. 
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No  longer  will  the  operationally  oriented  deeisions  be  the  sole  domain  of  a  few  highly 
experieneed  airmen.  While  SSLC2  will  not  make  the  deeisions  for  the  maintainer,  it  will 
present  all  the  information  needed  to  make  well  informed  deeisions  and 
ree  ommendations . 

Enhaneed  situational  awareness  provided  by  the  SSLC2  eoneept  ean  extend  aeross 
eommand  eehelons  and  provide  MAJCOM  and  Air  Foree  Forees  (AFFOR)  deeision- 
makers  visibility  into  wing-level  operational  eapabilities.  They  ean  look  down  into  the 
operating  loeations  and  determine  independently  if  the  resourees  are  in  plaee  to  meet  the 
flying  sehedule  and  Air  Tasking  Order  (ATO).  These  deeision  makers  ean  also  quiekly 
assess  the  impaet  of  any  degradation  of  sortie  eapaeity. 

SSFC2  is  the  logieal  eonelusion  to  meet  future  needs  of  the  Air  Foree.  The  SSFC2 
researeh  program  is  a  key  projeet  toward  faeilitating  informed  deeisions  at  multiple 
levels. 

2.3  SSLC2  SPIRALS 

The  SSFC2  program  had  three  Spirals.  Spiral  One  was  seoped  on  wing-level 
personnel  eoneentrating  on  one  primary  deeision,  the  fix/swap  deeision  of  an  aireraft. 
During  Spiral  One,  researehers  investigated  and  validated  the  eognitive  model  of 
Expeditors  on  the  flightline  and  eompared  eommereial  off-the-shelf  (COTS)  RFID 
teehnology  (WhereNet)  for  supporting  user  tasks  with  a  system  in  whieh  sensor 
information  was  integrated  with  flightline  information  (SSFC2).  The  results  indieated 
that  users  preferred  the  SSFC2  integrated  approaeh.  (See  Spiral  One  Final  Report  for 
more  details  [10]. 

Spiral  Two  was  used  to  enhanee  a  separate  program  eontraet  ealled  Virtual  Spaee 
Fogisties  Readiness  Center  (VSFRC)  as  a  modifieation  to  the  SSFC2  program  providing 
the  Air  Foree  Spaee  Command  (AFSPC)  and  Spaee  and  Missile  Systems  Center  (SMC) 
an  aetual  implementation  of  the  SSFC2  eoneepts.  Rather  than  using  RFID/RTFS 
teehnology,  VSFRC  fuses  data  from  numerous  AFSPC  sourees  to  provide  logisties  status 
of  spaee  ground  assets  to  determine  their  operational  eapability.  (See  Spiral  Two  Final 
Report  [11].) 
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Spiral  Three  eontinued  the  work  from  Spiral  One  but  expanded  the  researeh  to 
investigate  the  smart  sensors  and  the  fusion  of  information  for  multiple  tasks  and 
problems  that  oeeur  on  the  flightline  and  also  investigated  how  to  support  the 
eollaborative  nature  of  the  tasks  and  sharing  of  logistie  resourees.  Spiral  One  and  Spiral 
Three  were  direetly  linked  with  information  gathered  during  Spiral  One  related  to  the 
eognitive  work  models  and  user  feedbaek  used  as  input  to  Spiral  Three.  Originally,  an  in- 
depth  seientifie  study  with  users  aeross  several  airbases  was  planned  with  the  intention  of 
eolleeting  objeetive  human  performanee  data  as  well  as  user  opinions  to  meet  Key 
Performanee  Parameters.  Unfortunately,  during  the  Spiral  Three  phase,  the  entire  projeet 
had  to  be  re-seoped  due  to  time  and  eost  eonstraints. 

2.4  PROGRAM  SCOPE 

SSCL2  is  a  6.3  Advaneed  Teehnology  Demonstration  (ATD)  program  researehing 
ways  to  make  the  human  warfighter  more  agile,  produetive  and  smart  about  the  eritieal 
logistie  resourees  required  to  support  eombat  operations.  SSLC2  is  an  effort  to  make 
existing  systems  “smarter”  by  utilizing  data  eolleetion  sensor  teehnologies  and  fusing 
data  with  existing  information  systems  to  improve  Logisties  Command  and  Control 
deeision  making.  The  program  produees  researeh,  seienee  and  methods,  and  evaluates 
how  existing  teehnologies  ean  be  value-added  enhaneements  to  existing/fortheoming 
systems.  The  SSLC2  goal  is  to  researeh,  demonstrate,  and  position  for  transition, 
teehnology  eapabilities  that  ean  be  applied  to  and  integrated  into  existing  systems  to 
produee  logisties-based  operational  eapability  awareness.  Spiral  Three  demonstrated 
teehnologies  and  this  report  speeifies  requirements  that  enable  near  real  time  monitoring 
of  information  on  eritieal  aspeets  of  the  logisties  infrastrueture.  SSLC2  is  not  generating  a 
new  information  system,  but  generating  requirements  and  speeifieations  that  ean  be 
transitioned  to  an  existing  legaey  system.  Teehniques  and  tools  that  ean  best  enable 
logisties  deeision-makers  to  have  a  high-level  view  of  logisties  operations,  identify 
potential  problems,  and  make  proaetive  deeisions  at  various  nodes  within  a  logisties 
support  environment  are  identified  and  validated. 
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2.5  DOCUMENT  OVERVIEW 


The  Spiral  Three  Final  Report  presents  the  main  aetivities  assoeiated  with  the  SSLC2 
program  during  Spiral  Three.  Seetion  3  presents  the  Researeh  Approaeh  whieh  provides 
details  of  the  objeetives  and  the  six  step  proeess  used  to  reaeh  the  objeetives.  This  seetion 
presents  the  design  eoneepts,  the  final  interfaee  designs,  as  well  as  hardware  and  software 
arehiteetures.  Section  4,  Scientific  Study  Approach,  presents  the  details  of  the  study 
methodology  for  the  field  demonstration.  Section  5,  Results  and  Discussion  presents  the 
user  feedback  results  obtained  during  the  field  demonstration.  Section  6  presents 
Recommendations  and  Section  7  is  Conclusions  and  Appendix  A  defines  Acronyms  used 
in  this  report. 

3  Spiral  Three  Approach 

The  goal  of  SSLC2  Spiral  Three  was  to  investigate  ways  to  support  logistic  decision 
makers  in  complex  fiightline  operations  through  the  fusion  of  existing  fiightline 
information  with  smart  sensor  information.  The  focus  was  on  providing  support  to 
Expeditors  and  ProSupers  on  the  fiightline.  The  five  research  objectives  were  as  follows: 

Objective  1:  Develop  smart  sensors  for  logistics  operations. 

Objective  2:  Develop  a  method  for  determining  how  data  should  be  fused  for 
presentation  that  focuses  on  supporting  decisions  and  collaboration. 

Objective  3:  Obtain  user  opinions  of  deployed  sensors  in  the  field. 

Objective  4:  Develop  a  simulation  test  bed  and  evaluate  user  feedback  on  integrated 
data  fusion  decision  support  concepts. 

Objective  5:  Develop  recommendations  for  transition  to  existing  systems. 

In  order  to  meet  these  objectives  a  research  approach  was  defined  with  six  primary 
steps  as  listed  below  and  illustrated  in  Figure  1. 

1.  Determine  the  information  and  decision  requirements  that  are  needed  to 
provide  decision  makers  a  shared  vision  and  support  collaboration.  Work 
with  RFID  suppliers  to  develop  smart  sensors. 

2.  Develop  a  conceptual  design  for  an  interactive  visualization  to  present  sensed 
data  fused  with  fiightline  data. 

3.  Present  conceptual  design  ideas  to  SMEs  and  refine  concepts. 

4.  Develop  a  simulation  test  bed  to  provide  the  ability  to  demonstrate  concepts 
and  receive  feedback  from  system  users. 
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5.  Present  coneepts  during  a  field  demonstration  using  the  test  bed  and  reeeive 
feedbaek  from  systems  users. 

6.  Develop  reeommendations  and  requirements  (updated  from  reeommendations 
and  requirements  from  Spiral  One). 


Figure  1.  Research  Steps 


13 


As  illustrated  in  the  figure,  information  from  Spiral  One  related  to  Expeditor  and 
ProSuper  tasks  [10]  and  deeisions  as  well  as  requirements  were  used  as  input  to  Spiral 
Three.  As  mentioned  in  Seetion  2.3  of  this  report,  Spiral  Two  [11]  eoneentrated  on 
visualization  for  air  and  spaee  and  was  not  direetly  tied  to  SSLC2.  This  seetion  of  the 
report  (Seetion  3.0)  briefly  summarizes  Spiral  One  inputs  and  deseribes  steps  1-  4  of 
Spiral  Three.  Steps  5  and  6  are  deseribed  in  seetions  4.0  and  5.0  respeetively.  Output 
from  Spiral  Three  provides  direet  input  to  Spiral  Four. 

3.1  INFORMATION  AND  DECISION  REQUIREMENTS 

Fusing  information  from  sensors  with  fiightline  data  to  provide  shared  understanding 
among  personnel,  to  improve  deeision  making,  and  to  impaet  logistic  operations  requires 
presenting  the  information  in  such  a  way  that  the  user  can  understand  the  information  and 
act  on  it.  Development  of  intuitive  interactive  visualizations  requires  a  systematic 
approach  that  includes  a  focus  on  the  user,  understanding  what  information  is  required  to 
make  decisions  and  to  enhance  effectiveness.  The  visualizations  must  integrate 
information,  providing  the  right  information  at  the  right  time.  While  there  are  general 
guidelines  when  developing  visualizations,  there  are  no  specific  rules  that  are  guaranteed 
to  provide  useful  intuitive  visualizations.  The  most  successful  visualizations  require  a 
user-centered  approach  which  includes  a  requirements  gathering  stage  to  develop  a 
thorough  understanding  of  the  information  users  need,  how  they  perform  their  work  and 
the  decisions  they  make.  There  are  several  frameworks  described  in  the  literature  that 
focus  on  a  user-centered  approach.  Mayhew  [16]  describes  the  user-centered  lifecycle 
approach  for  the  development  of  software  systems.  There  is  the  Work  Centered  Support 
Systems  (WCSS)  concept  described  by  Eggleston  and  colleagues  [3,4].  This  concept 
focuses  on  the  development  of  systems  that  support  user  work.  Another  method  is  the 
Applied  Cognitive  Work  Analysis  (ACWA)  described  by  Paradis,  et  al.  [17]  and 
Gualtieri,  et  al.  [12]. 

All  of  these  frameworks  are  similar  in  that  they  are  focused  on  performing  up  front 
cognitive  work  analysis  to  understand  what  the  user  needs  and  transforming  the  data 
gathered  into  actual  design  of  user-interfaces  or  visualizations.  The  ACWA  framework 
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describes  the  need  for  mapping  the  data  and  information  into  a  visualization  for  the 
purpose  of  decision  support.  Even  with  these  concepts,  the  difficulty  lies  in  moving  from 
information  to  design.  No  framework  provides  guidelines  for  the  specific  design  of 
visualizations  because  visualizations  are  primarily  domain  and/or  work  dependent. 
Visualization  design  is  both  an  art  and  a  science,  requiring  creative  ideas  combined  with 
knowledge  of  human  capabilities  and  limitations  related  to  both  human  cognition  and 
visual  perception. 

In  addition  to  gathering  information  and  applying  it  to  design,  the  user-centered 
process  also  includes  evaluation  of  the  concepts.  Design  concepts  are  evaluated  early  and 
iteratively  during  the  process.  It  is  also  important  to  apply  knowledge  of  human 
perception  and  cognition,  guidelines  for  the  development  of  visualizations,  and  human- 
computer  interaction  guidelines.  For  example,  visualizations  should  be  designed  to 
minimize  the  need  for  users  to  rely  on  their  memory.  The  ability  to  provide  the  user  with 
situation  awareness  is  an  important  aspect  of  the  system.  For  this  application  the  system 
will  be  used  outdoors  and  therefore  display  contrast  and  luminance  output  capabilities 
and  color  choices  are  an  issue.  The  use  of  color  in  most  outdoor  applications  is  limited 
because  of  the  loss  in  contrast,  and  based  on  military  standards  we  must  have  redundant 
coding  if  color  is  used. 

3.2  SPIRAL  ONE  SUMMARY 

Spiral  One’s  objective  was  to  compare  the  use  of  commercial  off-the-shelf  (COTS) 
RFID  technology  (WhereNet)  for  supporting  user  tasks  with  a  system  in  which  sensor 
information  was  integrated  with  flightline  information  (SSFC2).  This  phase  helped  to 
gather  additional  system  requirements  for  future  phases.  To  develop  the  SSFC2  interface 
for  this  evaluation,  a  user-centered  approach  was  followed.  The  first  step  consisted  of 
systems  analysis  and  cognitive  work  analysis  (CWA)  with  data  collected  from  subject 
matter  experts  (SMEs)  at  various  Air  Force  bases  through  job  shadowing  and  interviews. 
The  second  step  focused  on  concept  development  and  storyboard  designs  with  SMEs 
participating  as  part  of  the  design  team.  The  third  step  developed  use  cases  from  the 
storyboards  to  create  a  simulation  of  the  interface  to  allow  for  user  testing.  The  focus  of 
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Spiral  One  was  to  support  one  deeision;  the  fix/swap  deeision.  The  flightline  information 
included  was  for  only  one  maintenance  day. 

The  SSLC2  general  interface  concept  is  illustrated  in  Figure  2  which  presents  the 
geographic  view.  The  interface  provided  the  user  with  aircraft  status  and  location,  daily 
flying  schedules,  personnel  and  equipment  resources  and  their  locations,  tasks  necessary 
to  fix  or  swap  aircraft,  time  to  complete,  time  to  move  resources  to  the  fix  site,  and  types 
necessary  to  complete  tasks,  specific  resources  assigned  to  tasks,  and  a  recommendation 
to  fix  or  swap  an  aircraft.  Users  could  also  view  the  information  using  a  flying  schedule 
view.  The  left  side  of  the  screen  presented  the  workflow  process  steps.  The  central 
portion  of  the  screen  provided  the  schedule  and  geographic  views.  The  views  could  be 
changed  from  an  overview  of  the  entire  flightline,  to  a  problem  view  focusing  on  one 
aircraft.  Users  could  interact  with  the  system  using  a  mouse  and  standard  keyboard. 


Context 


Selector 

Resources 


Problem 

Aircraft 


Tabs  Working  View 


[ 


Options 

Figure  2.  Spiral  One  Conceptual  Design 


3.2.1  User  Feedback  Spiral  One 

Fifteen  maintenance  personnel  from  four  airbases  interacted  with  the  interface  in  a 
simulation.  All  participants  had  ProSuper  and/or  Expeditor  experience.  A  great  deal  of 
data  was  collected  on  user  feedback  for  this  design  and  in-depth  details  can  be  found  in 
the  Spiral  One  Final  Report  [10]  and  in  Gallimore  et  al.  [8].  In  this  section  we  describe 
some  of  the  issues  noted  by  users  related  to  the  interface.  Participants  made  positive 
comments  about  the  ability  to  see  a  fiightline  overview,  both  geographically  and  as  a 
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schedule.  ProSupers  and  Expeditors  are  expeeted  to  multi-task  and  respond  to  more  than 
one  problem  at  a  time.  Partieipants  suggested  that  a  future  system  provide  them  the 
ability  to  multi-task  and  display  information  about  every  aireraft  in  order  to  support 
decision  making. 

SSLC2  also  broke  down  eaeh  problem  into  a  series  of  tasks  to  be  eompleted  (a  work 
centered  approaeh).  Eaeh  task  listed  assoeiated  equipment  and  personnel  resourees  with 
their  time  to  site  and  time  to  fix.  Participants  voiced  their  support  for  seeing  the  list  of 
tasks  and  said  they  would  use  this  feature.  Partieipants  also  said  they  would  prefer  the 
ability  to  create  or  modify  the  tasks  needed  to  fix  problems.  Participants  indicated  that 
having  the  entire  listing  of  tasks  and  resourees  was  overwhelming.  Participants 
sometimes  referred  to  the  wrong  task  when  trying  to  ehange  a  resouree,  and  indieated  too 
many  steps  were  required  to  realloeate  resourees  (up  to  eight).  Partieipants  wanted  a  eliek 
and  point  interfaee  rather  than  dialog  boxes  and  pull  downs.  Additional  items  and 
eomments  highlighting  positive  aspeets  of  SSEC2  functionality  included  the  ability  to 
zoom  in  and  out  within  the  geographieal  views.  The  results  indicated  that  users  preferred 
the  integrated  SSEC2  approaeh  to  the  off-the-Shelf  WhereNet  system  that  only  provided 
resouree  loeation  information  and  no  fiightline  information  [8,10]. 

3.3  DECISION  AND  INFORMATION  REQUIREMENTS  MAPPING 

ProSupers  and  Expeditors  make  many  deeisions  requiring  signifieant  eollaboration 
among  personnel  and  organizations.  Multiple  users  need  aeeess  to  shared  information  and 
it  must  be  provided  beyond  one  day.  Spiral  One  was  not  meant  to  taekle  all  these  issues, 
and  feedback  from  Spiral  One  was  important  for  Spiral  Three.  However,  it  was 
determined  that  to  ineorporate  multiple  aireraft  maintenanee  problems  and  to  inelude  all 
the  data  related  to  the  flightline  beyond  one  day,  the  Spiral  One  interface  would  not  be 
adequate.  There  were  too  many  menus  and  steps  to  get  to  important  information.  The 
interfaee  developed  in  Spiral  One  had  both  positive  and  negative  aspeets. 

The  user-eentered  approaeh  for  Spiral  Three  was  similar  to  that  used  in  Spiral  One, 
however,  additional  information  related  to  deeisions  made  and  information  needed  for 
Expeditors  and  ProSupers  was  gathered  in  Spiral  Three  by  reviewing  a  variety  of 
flightline  doeuments  from  multiple  El.S.  bases,  training  manuals,  and  working  with 
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SMEs.  SMEs  included  personnel  from  AERL  Eogistics  Researeh  Branch  as  well  as 
discussion  with  personnel  at  Seymour  Johnson  Air  Eorce  Base  which  was  considered  as  a 
possible  test  site.  Erom  this  data  specific  decisions  were  gathered.  The  decisions  were 
presented  to  SMEs  who  then  narrowed  down  the  decisions  to  thirty-one  whieh  are 
presented  in  Table  1. 


Table  1.  Decisions  Made  by  ProSupers  and  Expeditors 


Decision 

1 

How  can  1  schedule  the  maintenance  to  maximize  AC  availability? 

2 

How  can  1  schedule  the  maintenance  to  take  advantage  of  heavy  maintenance? 

3 

How  will  unscheduled  maintenance  affect  phase  flow? 

4 

How  should  1  schedule  CANN  rebuilds?  Affects  manpower. 

5 

Are  sufficient  personnel  resources  and  types  available  today  (now)? 

6 

Are  sufficient  equipment  resources  and  types  available  today  (now)? 

7 

Are  required  parts  available  for  maintenance  actions?  What  is  the  status  of 
backorders? 

8 

What  facilities  are  needed  and  available  for  current  maintenance  operations? 

9 

Can  scheduled  aircraft  meet  the  sortie? 

10 

Can  NMC  aircraft  be  repaired  in  time  to  meet  schedule  (in  a  line  or  as  a  spare)? 

11 

How  does  the  CANN/Hangar  queen  rebuild  affect  unscheduled  maintenance? 

12 

Are  sufficient  and  properly  configured  spares  available?  Are  they  configured  for 
takeoff? 

13 

Which  aircraft  problem  has  priority? 

14 

Must  aircraft  configuration  be  changed? 

15 

What  maintenance  actions  are  needed? 

16 

What  is  the  priority  of  repair?  E.g.  Might  be  PMC  and  can  make  next  flight  so 
priority  is  lower  than  another  PMC  or  NMC 

17 

What  servicing  is  required? 

18 

What  is  the  initial  assessment  of  the  problem? 

19 

Should  Cannibalization  be  used  to  solve  the  problem? 

20 

Should  the  problem  be  solved  by  swapping  the  AC? 

21 

Should  the  mission  be  delayed? 

22 

Should  the  mission  be  cancelled? 

23 

How  will  the  problem  affect/impact  the  flying  schedule? 

24 

What  are  the  causes  for  maintenance  deviations  (cancellations,  aborts,  additions  or 
early/late  takeoffs) 

25 

What  are  the  estimated  fix  times?  (How  long  will  it  take  to  fix,  configure,  CANN,  etc) 

26 

Are  specific  aircraft,  equipment,  systems,  or  subsystems  contributing  to  a 
disproportionate  share  of  deviations/turbulence? 

18 


Decision 

27 

Are  there  seasonal  or  cyclical  variations?  Are  current  variations  outliers? 

28 

What  factors  are  causing  an  increase  or  decrease  in  the  NMC  rates? 

29 

How  do  today's  problems  affect  my  schedule  tomorrow  given  different  maintenance 
issues  that  are  on  tomorrows  schedule,  next  week,  etc. 

30 

How  does  a  change  in  personnel  availability  changes  affect  my  schedules? 

31 

Which  task  that  I  personally  need  to  complete  has  priority?  (e.g.  monitor  a  take  off 
vs  request  something) 

In  addition  to  establishing  the  deeisions,  information  or  data  elements  that  are  used  on 
the  flightline  were  eomplied.  A  list  of  157  different  data  elements  was  ereated.  (See 
Appendix  B,  Data  Elements,  for  full  list). 

For  visualization  design  it  is  important  to  understand  how  the  data  elements  map  to 
the  deeisions.  Not  ah  information  is  needed  for  eaeh  type  of  deeision.  The  goal  was  to 
provide  users  with  easy  aeeess  to  data  based  on  deeisions  they  make.  Current 
frameworks  discuss  the  need  for  the  CWA  and  understanding  of  the  information  and 
decisions  to  be  applied  to  design,  but  moving  from  this  information  to  design  can  still  be 
difficult.  The  results  of  the  CWA  during  Spiral  One  were  an  important  input. 

3.3.1  Cluster  Analysis 

In  order  to  perform  a  mapping  of  what  information  is  needed  for  different  decisions 
and  to  determine  the  importance  of  information  we  decided  to  use  a  cluster  analysis 
technique.  A  data  matrix  was  created  with  the  31  decisions  in  columns  and  the  157  data 
elements  in  rows.  Three  SMEs  were  then  asked  to  read  each  decision  and  rate  the 
importance  of  the  157  data  elements  to  that  decision.  They  rated  the  information  on  a 
scale  of  1  to  5;  1-  not  at  ah  important,  2  -  somewhat  important,  3  -  important,  4  -  very 
important,  and  5  -  critical  or  extremely  important.  In  addition  to  the  31  decisions  they 
also  provided  a  rating  to  indicate  whether  the  data  element  should  be  visible  most  of  the 
time. 

The  ratings  were  used  as  input  for  the  cluster  analysis.  A  hierarchical  cluster  analysis 
using  Ward’s  method  was  conducted  within  the  JMP  Professional  statistics  package 
(Version  5. 0.1. 2).  The  ratings  were  first  averaged  across  the  three  SMEs.  The  cluster 
analysis  was  conducted  across  ah  decisions  to  be  made.  The  analysis  resulted  in  28 
clusters  of  the  157  data  items.  Appendix  B,  Data  Elements,  provides  a  summary  of  the 
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cluster  results  in  a  table.  The  table  ineludes  a  eolumn  which  provides  the  average  rating 
of  subjects’  responses  across  all  31  decisions. 

The  data  items  were  also  examined  by  the  average  rating  for  the  question  of 
“important  data  to  be  visible  most  of  the  time”.  The  seeond  column  of  the  Table  in 
Appendix  B,  Data  Elements,  provides  this  rating. 

Eighty-nine  of  the  157  data  items  were  given  an  average  rating  aeross  all  deeisions  of 
less  than  2  indieating  they  were  eonsidered  “not  at  all  important”  by  the  SMEs.  Table  2 
is  a  redueed  table  for  data  elements  that  received  average  ratings  of  2.0  (somewhat 
important)  or  above  across  all  decisions.  The  data  elements  in  this  table  are  ordered  from 
highest  rating  to  lowest  rating. 


Table  2,  Data  Elements  rating  2.0  or  higher 


Data  Element 

Rating 

Current  Discrepancies 

4.48 

Nature  of  problem. 

4.48 

Tail  number  (element  aircraft) 

4.38 

Reason  for  status 

4.23 

Tasks  required  to  solve  problem 

4.23 

Schedule  for  day  (Current  time  and  date,  all  aircraft,  Status,  brief  time,  take  off  time, 
land  time  duration  of  flight,  load,  remarks,  turn,  weather.  Phase?) 

4.19 

ETIC 

4.15 

Status  (PMC,  NMC,  FMC,  spare,  CANN) 

4.06 

Priority  of  task 

3.95 

Discrepancies  awaiting  parts 

3.58 

Scheduled  Maintenance 

3.57 

Safety/Danger  Considerations  (power/hydraulic  applications,  stress  panels 
removed,  aircraft  on  jacks,  weight  and  balance,  etc.) 

3.56 

MICAPS 

3.53 

Corrective  actions  (maintenance) 

3.49 

Personnel  current  a/c  working  or  task 

3.47 

Discrepancies  awaiting  maintenance 

3.38 

Personnel  Availability 

3.34 

Priority  of  sorties 

3.32 

take-off  time 

3.31 

Schedule  changes 

3.27 

Lines  required  for  filling 

3.10 
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Data  Element 

Rating 

Service  item  status,  requirement  for  next  flight 

3.10 

Status  of  backorder  parts 

3.03 

Supplies  in  stock,  Inventory  levels 

3.03 

Schedule  for  the  week 

2.91 

Availability  of  critical  equipment 

2.90 

Current  Configuration 

2.89 

Location  on  flight  line 

2.84 

Qualification  Levels  (3-level,  5-level,  7-level) 

2.81 

TCTO  (Time  Compliance  Technical  Orders)  -  Safety  Issues  -  Contains  A/C  Tail 
Number,  Date 

2.77 

OTI  (One  Time  Inspection)  Tail  Number,  Date 

2.76 

Personnel  qualifications,  certifications 

2.74 

Mission  Requirements 

2.73 

Land  time 

2.71 

A/C  Projected  to  be  Spares 

2.71 

A/C  Projected  to  be  Prime  Flyers 

2.71 

Weather  forecast 

2.70 

TCI  (Time  Change  Item) 

2.69 

MESL  (for  mission,  basic  and  full) 

2.65 

Personnel  role  (crew  chief,  specialist,  etc) 

2.63 

List  of  CANNED  items 

2.61 

Turn 

2.55 

Line  number  (details  type  of  sortie) 

2.55 

Type  of  a/c  (Model) 

2.49 

Type  of  sortie  on  schedules  (night,  XC,  training) 

2.48 

A/C  Projected  to  be  USM 

2.47 

Crew  chief  currently  assigned 

2.47 

Top  Supply  Drivers/Parts  that  have  problems 

2.42 

Work  Unit  Code 

2.42 

Shifts  Scheduled 

2.39 

Stage  of  CANN  jet 

2.38 

Past  week  maintenance  history 

2.33 

Load 

2.27 

AFSC 

2.23 

Personnel  Current  location 

2.22 

Rules  for  exceptional  releases 

2.20 

Maintenance  Training  Requirements 

2.20 
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Data  Element 

Rating 

Ground  Time  Requirements 

2.19 

a/c  load 

2.19 

Scheduling  guidelines 

2.18 

Days/time  since  last  flown  (hangar  queen) 

2.14 

Personnel  Contact  info  (name,  radio  call,  or  other) 

2.11 

Supplies  on  order  and  expected  delivery  date 

2.10 

Aircraft  with  pacer  motors  identified. 

2.05 

Equip  Availability 

2.05 

Work  schedules  (shift) 

2.03 

Equip  Current  status  (broke,  not  broke) 

2.03 

Scheduled  Maintenance  Requirements  (Generic  high  level  used  to  create  yearly 
forecast  -  example:  average  1  AC  out  per  month  for  depot ) 

2.01 

The  results  of  the  eluster  analysis  helped  designers  determine  grouping  of  information 
and  a  general  design  eoneept. 

3.4  CONCEPTUAL  DESIGN 

Based  on  output  from  Spiral  One  (CWA,  eoneept  maps,  field  test  results)  and 
eontinued  analysis  during  Spiral  Three  of  user  tasks,  deeisions,  data,  and  eluster  analysis, 
a  eoneeptual  design  was  developed.  Important  design  considerations  were  learned  from 
Spiral  One.  User’s  mental  model  of  flightline  maintenance  is  directly  tied  to  the  flying 
schedule  and  decisions  are  based  on  this  information;  therefore,  a  flying  schedule  is 
crucial.  Users  also  need  to  understand  maintenance  occurring  on  aircraft  and  the  concept 
of  seeing  the  specific  maintenance  subtasks  was  a  feature  that  users  liked  about  Spiral 
One.  Users  also  liked  the  concept  of  a  geographic  view  for  visualizing  where  resources 
are  located,  although  exact  locations  were  not  very  important  to  Expeditors  for  most 
equipment.  The  ability  to  locate  personnel  was  important  to  most  users  as  they  often 
spend  a  great  deal  of  time  searching  for  people.  Another  recommendation  from  the 
Spiral  One  report  was  that  the  SSLC2  interface  should  be  a  direct  manipulation  interface 
to  include  features  such  as  point  and  click  and  drag  and  drop  to  reduce  the  need  to  drill 
down  into  many  submenus  to  find  information.  Because  of  the  large  amount  of 
information  available  there  is  also  a  need  for  sorting  and  filtering  information  so  the  user 
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does  not  become  overwhelmed  and  can  easily  find  the  information  they  need.  Portability 
was  another  key  feature  as  Expeditors  are  usually  in  their  trucks  on  the  flightline. 

An  interactive  visualization  concept  was  created  to  fuse  the  data  and  keep  the 
information  in  logical  presentations  related  to  how  Expeditors  and  ProSupers  think  of, 
and  perform  their  work.  The  concept  was  to  provide  a  direct  tapping  interface  for  use  on 
a  Tablet  PC  and  provide  users  with  global  awareness  of  information  and  more  detailed 
relevant  information  as  needed  without  requiring  users  to  continuously  drill  down 
through  many  levels.  The  visualizations  should  allow  users  to  sort,  rearrange,  promote, 
focus,  filter  and  spotlight  data.  Users  must  be  able  to  make  notes,  highlight  information, 
and  collaborate  with  other  users.  Shniederman’s  [19]  concept  of  Overview  first,  zoom, 
filter,  sort,  and  details  on  demand  was  followed.  The  conceptual  designs  were  created  in 
Adobe  Macromedia  Plash  in  order  to  present  the  concepts  to  SMEs. 

The  conceptual  design  is  a  tab-based  interface  to  allow  users  to  easily  point  to  the 
needed  information.  The  primary  grouping  of  information  on  the  tabs  are:  flying 
schedule,  maintenance,  personnel,  equipment,  facilities,  fleet  health,  and  additional 
resources  (checklists,  AP  documents,  lookup  information).  This  Spiral  concentrated  on 
all  views  except  fleet  health  and  additional  resources.  Pigure  3  illustrates  the  general 
concept.  The  display  is  divided  into  four  primary  areas:  A  primary  (content  based) 
window,  a  geographic  (geo)  view,  a  detailed  data  view,  and  a  message  area.  The 
information  from  the  geo  and  detail  window  can  switch  places  with  the  primary  window 
so  that  a  larger  view  can  be  seen.  The  information  in  the  primary  window  always  drives 
the  information  to  be  displayed  in  the  detail  and  geo  views,  even  when  the  windows 
change  location.  Because  the  window  panes  are  limited  in  size  it  is  necessary  to  allow 
the  user  to  visualize  the  information  without  requiring  them  to  continuously  scroll. 
Therefore  the  concept  calls  for  the  use  of  zoomable  user  interface  (ZUI)  using  space- 
scale  techniques  [6]. 

Note  the  window  layout  for  conceptual  design  in  Pigure  3  is  not  drawn  to  scale. 
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3.4.1  Flying  Schedule 

Figure  4  illustrates  the  design  concept  of  the  flying  schedule  using  Adobe 
Macromedia  Flash.  The  purpose  of  this  view  is  to  show  the  current  day,  week  or  month 
of  scheduled  sorties  and  all  relevant  information  related  to  the  aircraft  and  sorties.  The 
frame  of  reference  is  a  time  line  with  schedules.  This  is  similar  to  the  concept  in  Spiral 
One.  This  view  requires  the  ability  to  chose  the  time  frame  by  which  to  work;  hours,  day, 
week,  etc,  with  the  user  in  control  of  how  much  of  the  schedule  to  view  at  one  time.  For 
example,  users  may  want  to  view  the  entire  week  at  once,  and  then  zoom  into  a  specific 
time  frame  within  that  week.  All  aircraft  that  are  overseen  by  the  Expeditor  are  visible 
on  the  schedule  view,  even  if  they  are  not  flying  on  a  specific  day.  The  user  is  able  to 
filter  and  sort  the  order  of  the  aircraft  presented  to  help  with  decluttering  and 
organization.  If  the  user  selects  an  aircraft  icon  on  the  left,  specific  details  related  to  that 
aircraft  are  shown  in  the  details  pane,  such  as  status,  phase,  turn,  etc.  The  aircraft  icon 
also  provides  information  in  the  way  it  is  color  coded  and  the  symbol  direction  providing 
redundant  coding.  If  the  user  highlights  the  icon  by  selecting  it,  the  location  of  the 
aircraft  on  the  fiightline  is  shown  in  the  geo  view. 

The  information  related  to  the  sortie  is  shown  on  the  time  line  with  important 
elements  such  as  milestones  (inspections,  crew  ready,  crew  show,  takeoff,  landing,  etc) 
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also  shown  on  the  timeline.  Additional  information  elements  important  to  users 
responsible  for  the  management  of  sortie  mission  sehedules  and  aireraft  preparation  are 
shown  below  the  sortie  time  line,  sueh  as  aireraft  eonfiguration,  load,  turn  time,  hours  to 
next  phase,  and  Estimated  Time  in  Commission  (ETIC).  Eaeh  of  these  items  is  seleetable 
in  order  to  bring  up  more  details  related  to  that  data  element.  In  eurrent  praetiee. 
Expeditors  and/or  ProSupers  write  down  the  time  when  events  oeeur  on  their  paper 
sehedules.  This  interactive  visualization  allows  the  milestone  markers  to  be  tapped  so  that 
the  computer  marks  the  time,  with  an  ability  to  edit  that  time  if  they  did  not  have  an 
opportunity  to  mark  the  event  in  real  time. 

Additional  functionality  from  this  screen  includes  the  ability  to  swap  an  aircraft. 
When  a  swap  is  recommended  by  the  Expeditor,  sent  to  and  approved  by  the  ProSuper, 
the  2407  form  required  for  this  change  is  automatically  filled  in  and  sent  the  Maintenance 
Operations  Center  (MOC). 
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3.4.2  Maintenance 

Monitoring  and  expediting  unscheduled  maintenance  is  a  primary  job  of  the 
Expeditor.  In  Spiral  One  the  Expeditors  were  provided  with  a  list  of  the  sub-tasks  needed 
to  complete  a  work  problem,  the  time  required  for  each  step  and  a  recommendation  for 
resources.  The  user  could  change  task  times  and  resources  and  the  system  would  provide 
new  time  estimates  based  on  the  time  to  get  the  new  resource  to  the  location.  However, 
the  users  indicated  that  the  time  changes  were  so  minor  that  they  did  not  consider  it 
necessary  unless  there  is  a  major  scheduling  conflict  with  resource  use.  However, 
conflicts  with  resource  use  were  not  included  in  Spiral  One.  Also,  the  time  estimate 
algorithm  was  based  only  on  straight  line  travel  which  is  not  feasible  for  the  flightline. 

Two  concepts  for  the  maintenance  view  used  to  display  information  about 
maintenance  tasks  and  resources  required  were  originally  considered;  a  list  concept  and  a 
diagram  concept.  Each  concept  is  briefly  described  below.  The  final  choice  for  the 
conceptual  design  was  the  diagram  concept. 

3.4.2. 1  List  Concept 

The  list  concept  is  to  display  a  detailed  listing  via  textual  presentation  of  all  necessary 
subtasks  and  required  resources  needed  to  fix  an  aircraft.  The  advantages  of  this  concept 
are  that  user  can  see  a  list  of  tasks  and  resources  needed  for  each  task  together  and  in 
detail  and  that  the  list  might  be  preferred  because  it  is  similar  to  how  they  read  their 
current  maintenance  schedule.  However,  the  disadvantages  are  that  it  requires  a  lot  of 
reading,  does  not  provide  the  user  with  global  situation  awareness,  and  does  not  allow  the 
user  to  quickly  glance  at  information  to  determine  if  some  specific  issues  have  arisen. 
The  list  requires  substantial  scrolling.  It  is  also  difficult  to  compare  resource  needs  across 
aircraft. 

3.4.2.2  Diagram  Concept 

The  purpose  of  the  diagram  concept  is  to  provide  a  global  visualization  that  allows 
the  user  to: 

•  Receive  an  overall  impression  and  be  able  to  perform  quick  assessment  of  the 
tasks  involved  to  fix  aircraft; 

•  Visualize  the  flow  and  timing  of  tasks,  including  those  that  can  occur 
simultaneously; 

•  Determine  if  there  are  any  resource  conflicts  on  a  task;  and 
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View  multiple  aircraft  at  the  same  time. 


Figure  5  illustrates  the  concept.  The  frame  of  reference  is  a  vertical  time  scale  and 
the  ability  to  display  multiple  aircraft  horizontally.  The  subtasks  required  to  complete  a 
maintenance  task  are  presented  as  a  flow  diagram,  flowing  vertically  from  top  to  bottom 
in  line  with  the  tasks  start  time.  The  time  line  is  shown  on  the  left  side  of  the  display.  The 
user  can  select  the  time  frame  by  which  to  view  the  information  using  zooming  functions. 
Current  time  is  marked  on  the  time  line.  Each  subtask  is  specified  by  a  box  that  is  broken 
into  rows  that  provide  quick  details  related  to  the  tasks  including  resources  assigned.  For 
example,  the  text  indicating  type  of  personnel  needed  is  included.  The  user  may  zoom  in 
on  subtasks  to  see  more  detailed  information.  When  the  user  puts  his/her  cursor  over  the 
subtask  box,  it  expands  in  size  so  that  all  relevant  information  can  be  read.  The  other 
boxes  are  reduced  in  size  but  still  be  visible  to  show  the  layout  of  the  task  sequence  (e.g. 
zooming  techniques  such  as  Fisheye).  If  the  user  selects  a  row  on  a  box,  the  pertinent 
information  is  displayed  in  the  details  view.  Completed  tasks  are  highlighted  (or  grayed 
out).  Areas  of  concern  are  highlighted  (color  coded)  within  each  box  when  there  is  a 
potential  conflict  or  if  the  task  is  running  late.  With  this  concept,  users  can  glance  at  the 
tasks  and  quickly  determine  if  there  is  an  issue,  such  as  a  conflict  or  overdue  task,  and 
concentrate  their  efforts  on  problem  solving. 

Currently,  Expeditors  make  notes  related  to  maintenance,  so  the  maintenance  view 
also  provides  the  ability  to  write  and  read  notes.  This  view  allows  the  user  to  make 
changes  to  the  tasks  in  terms  of  start  and  end  times  and  resources  assigned.  Some 
possible  data  entry  methods  to  consider  include  drag  and  drop  techniques  to  allow  the 
user  to  change  start  and  end  times,  clicking  on  the  individual  rows  to  allow  users  to 
change  resources  and  make  notes,  or  clicking  on  a  subtask  to  open  a  window  where  all 
subtask  data  can  be  entered  simultaneously. 

The  advantages  of  this  technique  are: 

•  Provides  an  overall  impression  and  quick  assessment  of  the  tasks  involved  to 
fix  aircraft. 

•  Provides  an  indication  of  the  most  efficient  task  flow  (including  tasks  that  can 
be  done  simultaneously). 
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•  Provides  quick  visibility  of  any  conflicts  that  must  be  considered  related  to 
resources. 

•  Ability  to  view  multiple  aircraft  at  the  same  time  for  comparison. 

•  Provides  support  for  ability  to  balance  multiple  tasks  and  resources  against 
multiple  problems. 

•  Provides  situation  awareness  of  multiple  aircraft  but  allows  user  to  reach  more 
detailed  information  on  demand. 

•  Relies  on  the  use  state  of  the  art  visualization  techniques,  such  as  ZUI  and 
fisheye  lens,  to  keep  the  user  aware  of  where  they  are  in  the  problem. 

The  Disadvantages  of  this  technique  are; 


•  Requires  drill  down  into  information. 

•  Designers  must  be  careful  in  design  of  coding  schemes. 

Based  on  the  advantages  of  the  diagram  concept,  it  was  selected  as  the  conceptual 

design  to  be  presented  to  SMEs.  Figure  5  depicts  an  example  of  this  design  concept  for 
the  maintenance  subtask  view  showing  tasks  and  subtasks,  drawn  using  Adobe 
Macromedia  Flash. 
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To  enable  the  user  to  determine  whieh  aireraft’s  maintenance  tasks  to  view  in  the 
subtask  maintenance  screen,  an  overview  of  all  maintenance  tasks  is  needed.  The 
maintenance  overview  screen  (see  Figure  6  drawn  using  Adobe  Macromedia  Flash)  is  a 
list  of  tasks  for  each  aircraft.  The  user  may  select  the  specific  tasks  they  want  to  view  on 
the  subtask  screen  by  checking  the  boxes  next  to  the  tasks  and  expanding  the  selected 
items  at  which  point  the  flow  diagram  of  the  subtasks  are  shown.  This  screen  also  shows 
coding  related  to  any  problems  that  may  be  occurring  in  a  maintenance  task  including 
conflicts  and  overdue  tasks.  In  this  way  the  user  can  concentrate  their  efforts  on  those 
tasks  which  show  issues  and  obtain  more  specific  details,  yet  see  a  list  of  all  tasks 
currently  required  on  an  aircraft  (scheduled  and  unscheduled). 
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Figure  6.  Overview  of  maintenance  tasks  concept 


3.4.3  Equipment 

Expeditors  primarily  rely  on  others  (e.g.  AGE  yard)  to  find  and  bring  equipment  to 
the  aircraft  being  maintained.  In  some  cases  when  they  have  an  immediate  need  they 
search  for  equipment  on  the  fiightline,  pick  it  up  with  their  truck  and  bring  it  to  the  work 
site.  They  may  then  realize  it  is  not  functioning.  It  is  not  unusual  for  Expeditors  or  others 
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to  be  looking  for  a  specific  resource.  The  purpose  of  the  equipment  view,  shown  in 
Figure  7  drawn  using  Adobe  Macromedia  Flash,  is  to  provide  the  user  with  a  global 
awareness  of  resource  availability  as  well  as  more  detailed  information  if  needed.  The 
frame  of  reference  is  the  specific  equipment  resources  and  their  current  status  as  well  as 
future  status  information  related  to  assignments  to  future  tasks.  For  example  if  a  piece  of 
equipment  is  assigned  to  a  task  at  a  specific  time,  this  information  is  shown  in  the  details 
view.  The  concept  is  to  allow  users  to  quickly  glance  at  equipment  to  determine  status 
(e.g.  broken,  in  conflict,  available).  If  they  notice  that  many  of  their  low  density  high 
demand  resources  are  broken,  that  knowledge  allows  them  to  consider  how  resource 
availability  may  affect  their  maintenance  so  they  can  plan  accordingly.  It  also  allows 
them  to  easily  locate  a  piece  of  equipment  on  the  fiightline.  Equipment  is  displayed  on 
the  primary  view  using  icons  and  redundant  color  and  shape  coding  to  designate  their 
status.  If  they  need  specific  details,  such  as  location  or  tasks  to  which  the  equipment  are 
assigned,  they  can  tap  on  the  equipment  and  see  the  details  in  the  detail  pane  and  the 
equipment  is  shown  in  it’s  current  location  based  on  RFID  tags  on  the  fiightline.  Users 
are  able  to  sort,  filter  and  zoom  on  this  view,  an  important  requirement  given  the  large 
number  of  possible  equipment  resources. 
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Figure  7.  Equipment  view  concept  design 


3.4.4  Personnel 

Personnel  are  also  resourees.  In  Spiral  One,  Expeditors  indieated  that  finding  people 
was  a  very  time  eonsuming  task.  The  eoneept  for  the  personnel  view  is  similar  to  the 
equipment  view;  however,  the  frame  of  referenee  is  a  sehedule  with  a  time  line.  Aetive 
personnel  tags  on  badges  eould  provide  loeation  as  well  as  availability.  The  personnel 
view  displays  the  person’s  name  on  the  left  and  their  sehedule  on  the  right.  Like  the 
flying  sehedule,  this  view  requires  the  ability  to  chose  the  time  frame  by  which  to  view 
the  information  (hours,  day,  week,  etc.)  with  the  user  in  control  of  how  much  of  the 
schedule  to  view  at  one  time.  The  user  can  filter  and  sort  in  order  to  declutter  and 
organize  the  data  and  only  show  information  of  interest,  or  by  work  group  (e.g.  view  all 
crew  chiefs).  Users  can  also  search  by  name.  Selecting  the  personnel  icon  on  the  left 
places  detailed  information  related  to  that  person  in  the  details  pane  and  shows  the 
location  of  the  individual  on  the  geo  view.  The  schedule  indicates  what  tasks  personnel 
are  assigned  and  also  conveys  if  they  have  been  assigned  to  multiple  tasks  at  the  same 
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time  causing  a  conflict.  Figure  8  depicts  the  conceptual  design  for  the  Personnel  view, 
drawn  using  Adobe  Macromedia  Flash. 

BOl 
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Figure  8.  Personnel  view  concept  design 


3.4.5  Facilities 

Figure  9  illustrates  the  design  concept  for  facilities,  drawn  using  Adobe  Macromedia 
Flash.  Maintenance  may  occur  on  the  flightline  at  various  facilities.  The  frame  of 
reference  for  this  view  is  a  schedule  view  which  shows  the  aircraft  assigned  to  the  facility 
and  the  time.  Details  related  to  the  facility  or  schedules  are  available  in  the  details 
screen.  Users  may  also  schedule  their  aircraft  to  a  facility  from  this  view. 
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Figure  9.  Facilities  view  concept  design 


3.4.6  Fleet  Health 

Our  research  has  shown  a  need  for  providing  data  related  to  fleet  health.  Therefore 
visualization  for  this  data  would  be  important.  However,  based  on  time  and  cost 
constraints  Spiral  Three  did  not  focus  on  these  visualizations. 


3.4.7  Additional  Resources 

Expeditors  refer  to  a  variety  of  manuals  and  resources.  The  purpose  of  this  function  is 
to  provide  a  place  in  which  to  open  and  view  electronic  resources  that  may  be  needed  in 
the  course  of  the  day.  Spiral  Three  did  not  focus  on  the  design  of  this  functionality. 


3.4.8  Collaboration 

Expeditors  must  collaborate  with  many  organizations  and  individuals.  The  area  below 
the  primary  window  is  a  communication  area  in  which  the  user  is  able  to  view  messages 
and  alerts.  The  design  concept  also  includes  the  functionality  of  allowing  the  user  to 
write  and  send  messages.  Additionally,  automated  messages  are  used  to  free  the  user 
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from  having  to  send  messages  manually  when  resourees  are  requested.  Collaboration  is 
also  enhaneed  by  providing  different  users  with  a  shared  vision.  The  SSLC2  system  ean 
be  viewed  by  Expeditors,  the  ProSuper,  MOC,  AGE,  and  any  others  who  need  insight 
into  the  maintenanee.  Each  user  may  have  a  different  view.  Eor  example,  the  ProSuper 
has  access  to  view  all  aircraft  whereas  each  Expeditor  views  the  aircraft  for  which  they 
are  responsible.  All  users  see  all  resources  for  sharing  and  awareness  of  resource  use. 

3.5  SUBJECT  MATTER  EXPERT  FEEDBACK 

The  conceptual  designs  briefly  described  and  illustrated  in  Adobe  Macromedia  Elash 
were  presented  to  four  SMEs  during  a  user  group  meeting.  The  concepts  were  explained 
and  SME  feedback  solicited.  In  general  the  SMEs  indicated  that  the  concepts  were  useful 
and  felt  that  we  should  proceed  to  detailed  design  on  the  concepts  presented.  They  also 
provided  additional  useful  information  which  is  summarized  in  Table  3. 


Table  3.  Feedback  from  SME  User  Group  Meeting 


Flying  Schedule  and  specific  aircraft  information 

Modifications  on  configurations  (Specific  details). 

Frag  list  with  load  plan. 

Pilot  crew  names  to  show  VIPs  and  commanders. 

Exceptional  Releases  -  prior  to  flight.  Add  to  line  of  schedule  instead  of  inspect.  (Also  known  as 
crew  ready.  Use  crew  ready  instead  of  exceptional  release. 

Could  also  add  de-icing  on  the  time  line. 

Static  displays  (aircraft)  on  the  flight  schedule  used  for  training  (e.g.  practice  loading)  or  to  show 
visitors. 

Maintenance 

When  changing  ETIC  on  the  maintenance  tasks,  do  not  change  it  on  the  flying  schedule.  Don’t 
want  to  keep  bumping  ETIC  all  the  time. 

In-process  inspections  should  be  identified  in  tasks  and  reminded. 

Equipment 

Show  what  task  the  equipment  is  assigned  to  along  with  scheduled  time  of  usage. 

Show  what  shop  the  equipment  belongs  to  (who  owns  it). 

Personnei 

Manning  number  (5  digit  unique  ID  for  each  person). 

Allow  the  assignment  of  tasks  under  personnel,  then  automatic  feedback  to  maintenance  screen. 

Faciiities 

Show  who  specifically  scheduled  the  use  of  the  facility. 
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General 

Want  to  see  original  flying  schedule  as  contracted  to  be  able  to  show  differences  to  what  was 
contracted  versus  what  was  changed. 

2407  schedule  change-  line  swap  or  drop  a  line  must  request  a  change  to  schedule  using  2407 
form. 

Need  another  tab  for  discrepancies  based  on  aircraft.  All  discrepancies  not  just  those  being  fixed 
now. 

Must  identify  who  made  changes  if  they  are  made  by  other  team  members. 

Add  to  specifications,  how  to  use  history  data. 

Checklists  are  needed.  (This  would  be  available  under  additional  resources) 


3.6  SIMULATION  TEST  BED 
3.6.1  User  Interface 

After  review  of  the  eonceptual  design  with  SMEs,  the  next  step  was  detailed  design. 
The  SSLC2  user  interface  detail  design  required  specifying  navigation,  specific 
information  on  each  screen,  icons,  color  usage,  redundant  coding,  error  messages,  etc. 
This  detailed  design  information  was  written  into  Use  Cases  and  a  simulation  test  bed 
was  then  programmed.  Of  the  twenty-two  (22)  use  cases  designed,  sixteen  (16)  were 
fully  implemented  and  one  (1)  was  partially  implemented. 

The  results  from  the  cluster  analysis  were  used  to  map  the  specific  data  elements  to 
the  interface.  The  Table  in  Appendix  B,  Data  Elements,  shows  the  mapping  of  the 
specific  data  elements  described  in  Section  3.1  to  their  placement  in  the  interface. 

Based  on  time  and  cost  constraints  it  was  not  possible  to  include  all  functionality 
specified  in  the  conceptual  design  and  Use  Cases  in  the  simulation  test  bed.  Also,  some 
functionality  of  the  simulated  interface  was  designed  in  a  way  that  was  easiest  to 
program.  Eor  example,  dialog  boxes  were  used  for  most  data  manipulation  although  a 
more  direct  drag  and  drop  interface  was  originally  conceived  and  should  be  considered  as 
a  requirement  in  future  system  updates.  The  2407  form  used  to  make  an  aircraft  schedule 
change,  was  also  not  implemented,  yet  the  approval  process  functionality  was 
implemented.  The  functionality  of  zooming,  sorting,  and  filtering  were  not  provided  in 
this  Spiral  but  are  considered  critical  requirements.  Eor  this  Spiral,  the  system  required 
scrolling.  Although  not  all  functionality  was  available,  the  simulation  test  bed  design  was 
such  that  the  users  were  able  to  understand  the  design  concepts  and  functionality  and 
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interact  with  the  system  to  provide  feedback  for  future  design  refinement  and 
specification  of  requirements.  Live  system  interfaces  were  also  not  feasible,  so  data  was 
extracted  from  these  systems  and  populated  into  SSLC2. 

3.6.2  Simulation  Test  System  Development 

The  purpose  of  the  simulation  test  system  was  to  provide  the  ability  to  test  as  well  as 

demonstrate  SSLC2  concepts.  This  section  briefly  details  the  overall  software 
architecture  followed  by  more  detailed  information  including  software  reuse  from  Spiral 
One,  Spiral  One  feedback,  use  case  specifications,  requirements  and  model  traceability, 
activity  and/or  design  model,  algorithms  and  business  logic,  flash  prototype  and  database. 

3.6.2. 1  Software  Architecture 

SSLC2  is  a  three-tier  thin  client  web  application.  The  client  tier  is  a  thin  client  web 
application  that  incorporates  the  GUI.  The  only  commercial  software  requirement  for  the 
client  machine  is  Microsoft  Internet  Explorer  version  5  or  greater  and  a  Java  Virtual 
Machine  version  1.4  or  greater.  All  the  GUI  objects  are  downloaded  to  the  client  the  first 
time  the  computer  accesses  SSLC2.  Subsequent  client  requests  result  in  download  of  data 
only.  This  design  reduces  the  bandwidth  requirements  and  provides  a  responsive  system. 

The  middle  tier,  the  business  logic  of  SSLC2,  is  comprised  of  servlets  currently 
developed  and  hosted  on  Apache  Tomcat  version  5.5;  however,  SSLC2  could  be  rehosted 
to  run  on  any  web  server  that  supports  Java  server  pages,  servlets  and  applets. 

Communication  to  the  tier,  the  database,  is  via  industry  standard  Java  Database 
Connectivity  (JDBC)  protocol.  The  database  used  is  Oracle  version  9i  and  can  be  hosted 
on  the  same  server  as  the  middle  tier  web  server  or  on  a  separate  server. 

The  only  other  software  interface  created  in  Spiral  Three  was  an  interface  to  the 
WhereNet  server  which  provides  the  simulated  resource  location  data.  The  WhereNet 
server,  a  separate  server  from  the  SSLC2  server,  supports  communications  via  HyperText 
Transport  Protocol  (HTTP).  A  server  side  process  was  created  that  subscribes  to  events 
on  the  separate  WhereNet  server  via  HTTP,  in  order  to  receive  location  information. 
This  information  is  retrieved  by  SSLC2  by  pinging  the  WhereNet  Application 
Programming  Interface  (API)  at  a  timed  interval  and  then  inserted  into  the  SSLC2 
database  via  JDBC. 
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3.6.2.2  Spiral  One  Reuse 

Capitalizing  on  the  previous  SSLC2  Spiral  One  we  were  able  to  produee  a  thin  elient 
web  applieation  prototype  for  the  Spiral  Three  Field  Test.  This  seetion  discusses  the 
major  software  components  of  Spiral  One  reused  in  SSLC2. 

A  major  reuse  of  Spiral  One  work  included  the  WhereNet  interface  and 
communication  functionality.  These  elements  included:  database  tables,  servlets,  data 
objects,  and  the  communications  object  used  to  transmit  data  to  and  from  the  server. 
Some  reuse  of  portions  of  the  schedule  occurred  as  well;  however  it  was  necessary  to 
redesign  the  framework  into  a  3 -panel  layout  with  links  and  controls  between  the  3- 
panels  and  use  animations  for  viewing  simplification.  Additional  enhancements  to  the 
schedule  included  increased  complexity  and  detail  to  include  new  icons  for  an  improved 
look  and  feel.  Interview  methods  were  used  to  collect  the  detail  required  by  the 
ProSuper,  Expeditor,  and  MOC.  This  resulted  in  researching  several  legacy  systems  for 
required  data.  Patriot  Excalibur  (PEX)  was  the  major  legacy  we  used  for  data  mining. 

The  same  geo  pane  from  Spiral  One  was  used  but  with  a  new  map  for  the  Toledo 
ANG  flight  line,  hangar,  AGE  yard,  avionics  shop,  hush  house,  and  engine  shop.  Eor  the 
detailed  view  we  reused  the  basic  concepts  of  display  with  much  improved  capability  and 
control  of  the  data.  The  SSEC2  team  reused  the  concept  of  Roles  and  incorporated  major 
improvements  in  identifying  three  major  users  with  variability  defined  rules  of  access. 

The  build  environment  included  batch  files  and  Apache  Ant  build  scripts  that 
compiled  the  source  files  and  created  a  deployment  directory  containing  the  entire  web 
application.  This  directory  was  controlled  through  configuration  management  process 
along  with  a  source  code  control  tool  called  Concurrent  Version  Systems  (CVS).  This 
allowed  for  a  simple  “drag-and-drop”  development  method  to  deploy  updates  to  the 
SSEC2  application. 

Generally,  the  decision  to  reuse  Spiral  One  elements  was  based  on  SSEC2 
functionality  and  research  requirements.  The  development  approach  used  the  Spiral  One 
basic  framework  and  reintroduced  elements  meeting  SSEC2  needs. 

3.6.2. 3  Use  Case  Specifications 

A  use  case  describes  a  sequence  of  actions  a  system  performs  yielding  an  observable 
result  to  a  particular  actor.  A  series  of  use  cases  were  created  describing  how  the 
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application  functions,  including  business  rules,  sereen  requirements,  user  roles,  and 
aceess  rules.  SSLC2  interaets  with  the  Expeditor,  Pro  Super,  and  MOC.  The  use  eases 
describe  eaeh  use  case  related  to  the  SSLC2  design  model.  The  realizations  are  presented 
through  sequence  diagrams.  A  sequence  diagram  is  a  structured  representation  of 
behavior  as  a  series  of  sequential  steps  over  time.  These  sequenee  diagrams  depiet  the 
work  flow  of  flightline  maintenance,  eommunieation  and  management  of  elements  in 
eooperation  over  time  of  maintenanee  tasks  and  schedules.  Use  cases  in  general  also  give 
a  basic  description,  flow  of  events,  alternative  flows,  speeial  requirements,  preconditions, 
post  conditions  and  any  additional  actors.  Table  4  lists  all  the  use  eases  developed  for 
SSLC2  Spiral  Three  and  identified  whether  eaeh  was  fully  implemented,  partially 
implements,  or  not  implemented  and  on  hold  for  a  future  Spiral. 


Table  4.  Spiral  Three  Use  Cases 


Use  Case 

Fully  Implemented 

Partially 

Implemented 

Not 

Implemented 

Login 

X 

Screen  Overview 

X 

Aircraft  Iconography 

X 

Alerts  and  Communication 

X 

Geo  View 

X 

Sort 

X 

References 

X 

Flying  Schedule  View 

X 

Update  Aircraft  Details 

X 

Swap  Aircraft 

X 

Ops  Schedule  Discrepancy 

X 

List  Maintenance  Task  Overview 

X 

Maintenance  Subtask  View 

X 

Maintenance  Details 

X 

List  Equipment  View 

X 

Update  Equipment  Details 

X 

View  Personnel  Schedule 

X 

Update  Personnel  Details 

X 

List  Facilities  Schedule 

X 

Maintain  Facilities  Schedule  and 
Details 

X 

38 


Use  Case 

Fully  Implemented 

Partially 

Implemented 

Not 

Implemented 

Update  Facilities  Detail 

X 

WhereNet  Interface  for  Location  of 
Resources 

X 

3.6.2.4  Requirements  and  Model  Traceability 

Essential  to  documenting  the  SSLC2  requirements  is  the  capability  to  trace  from  the 

source,  such  as  a  literature  search  documents,  interviews,  previous  research  efforts, 
through  the  use  cases  to  the  software  components  and  test  cases.  The  Spiral  One  Final 
Report  was  used  along  with  the  statistical  cluster  analysis  results,  and  other  decision 
support  research  to  create  visualization  and  interface  requirements  as  input  into  the  use 
cases.  Use  case  specifications  were  developed  from  these  requirements  and  linked  to 
specific  software  components  and  then  test  cases. 

3.6.2. 5  Activity  and/or  Design  Model 

For  Spiral  Three,  SSFC2  begins  when  a  problem  is  already  identified,  assessed  and 
verified.  Task  steps  for  fixing  or  swapping  the  aircraft  and  times  and  types  of  resources 
are  known  and  in  the  system.  SSFC2  assigns  resources  to  the  problem  aircraft  based  on 
business  rules  related  to  priority,  proximity,  availability,  certification,  and  ownership. 
Using  SSFC2  the  Expeditor  monitors  the  flightline  and  is  alerted  when  a  problem  is 
encountered  by  the  system.  The  message  or  alert  is  detailed  in  what  the  problem  is  and 
along  with  task,  tail  number  and  resource  affected.  The  Expeditor’s  first  step  is  to  make  a 
choice  in  how  to  view  the  problem  through  the  applications  multiple  views.  He  can  view 
information  through  the  flying  schedule  view,  the  maintenance  views,  the  equipment 
views,  the  personnel  views,  the  geo  view,  or  the  facility  view.  The  Expeditor  reviews  the 
resources  assigned  by  SSEC2  for  maintenance  tasks  and  both  the  fix  and  the  swap 
solutions.  The  Expeditor  can  override  recommendations,  select  any  additional  resources 
or  manipulate  times  based  on  experience. 

The  scenario  concludes  with  the  Expeditor  selecting  a  course  of  action.  His  options 
are  to  fix  or  swap  the  aircraft  or  delay  or  cancel  the  sortie.  If  fix  is  selected,  the  SSEC2 
concept  is  to  automatically  notify  resource  managers,  technicians,  and  inform  the 
ProSuper  and  MOC.  If  a  swap  is  selected,  it  is  considered  a  recommendation  that  goes  to 
the  ProSuper  for  approval.  The  final  option  is  to  recommend  delay  or  cancel.  This  also 
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becomes  a  recommendation  to  the  ProSuper  who  in  turn  negotiates  a  course  of  action 
with  Operations.  In  SSLC2,  the  problem  information  as  well  as  the  potential  solutions 
would  be  presented  to  the  ProSuper  to  facilitate  the  negotiations. 

The  User  Interface  contains  50  screens  which  includes  both  primary  and  secondary 
screens.  Filter  and  Sort  abilities  were  included  in  most  of  the  screens  for  viewing  ease. 
Animations  were  performed  using  applets  with  the  communication  link  to  a  servlet  on  the 
SSLC2  Server  through  port  80. 

3.6.2. 6  Algorithms  and  Business  Logic 

The  SSLC2  simulation  uses  numerous  algorithms  to  determine  resource  availability, 
resource  selection,  and  which  aircraft  to  swap.  The  logic  for  these  algorithms  is 
described  below. 

1.  Resource  Availability  -  The  specific  location  of  resources  was  specified  in 
WhereNet.  In  this  system  it  is  possible  to  define  zone  labels  within  the  RTFS 
coverage  area.  The  zone  information  is  encoded  within  the  RFID  tag  and  sent  in 
the  message  to  SSLC2.  Zones  were  defined  for  the  entire  flightline,  each  parking 
location,  AGE  yard,  ready  lines,  etc.  In  order  for  SSLC2  to  consider  a  resource 
as  being  available  and  therefore  recommended,  it  had  to  be  FMC,  equivalent  to 
the  resource  it  is  replacing,  and  in  any  available  zone.  Resources  in  any  other 
location  were  available  to  be  manually  selected. 

2.  Candidate  Swap  Aircraft  -  SSLC2  not  only  recommends  resources  to  meet  a  fix 
requirement  but  also  recommends  resources  to  meet  a  tail  swap  requirement. 
One  of  the  resources  needed  is  a  candidate  tail  for  the  swap.  SSLC2  used  the 
following  logic: 

•  If  there  is  an  aircraft  designated  as  a  spare,  in  the  same  configuration 
and  on  the  flying  schedule  then  that  is  the  candidate  swap; 

•  If  there  is  not  a  spare  designated,  the  candidate  tail  is  the  last  flyer  on 
that  day  with  the  same  configuration;  or 

•  If  no  other  flyers  are  available,  then  select  an  FMC  aircraft. 
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3.  System  and  Automated  Alerts  -  Multiple  algorithms  were  ineluded  related  to 
suggestive  alerts  on  issues  and  problems  related  to  resouree  ehanges,  aireraft 
swaps,  and  ETICs. 

4.  Resource  availability  -  Multiple  algorithms  were  included  to  compute  resource 
availability,  equivalency,  schedule  availability,  certification  matches,  and 
organization  approvals. 

3.6.2. 7  Database 

Some  of  the  database  design  was  inherited  as  a  bi-product  of  reuse  of  Spiral  One. 
There  were  some  significant  changes  from  Spiral  One  in  order  to  accommodate  Spiral 
Three’s  complexity  and  design.  Changes  included  design  elements  such  as  Roles  and 
User  tables,  storing  very  detailed  equipment/personnel  information,  aircraft  problems  and 
configurations,  and  organizations  that  relate  to  the  fix  or  swap  decision.  The  aircraft 
problem  requirement  generated  the  need  to  be  able  to  associate  tasks,  locations,  resource 
details,  organization,  availability  and  times  to  both  the  fix  and  swap  options. 

The  roles  concept  equates  to  a  specific  set  of  software  capabilities  that  are  available  to 
different  types  of  users.  The  capabilities  relate  to  the  types  of  views  the  user  is  able  to 
access  and  also  authorization  related  to  approval. 

The  Spiral  One  database  was  designed  to  handle  simple  relationships  between 
aircraft,  equipment,  personnel,  and  maintenance  tasks.  Spiral  Three  expanded  on  this 
concept  by  adding  very  detailed  information  which  emulated  legacy  information 
available  in  such  systems  as  Patriot  Excaliber,  CAMS,  and  G081. 

4  Scientific  Study  Approach 

Spiral  Three  was  originally  scheduled  to  include  a  research  study  to  obtain  objective 
user  performance  as  well  as  subjective  opinions  on  the  use  of  enhanced  sensors  fused 
with  information  from  the  fiightline  to  support  maintenance  tasks,  collaboration  and 
decisions.  However,  due  to  budgeting  constraints  the  scientific  study  was  redesigned  as  a 
field  demonstration  at  one  location.  This  section  describes  the  approach  for  the  field 
demonstration. 
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4.1  OBJECTIVE 

The  objective  of  this  study  was  to  demonstrate  SSLC2  and  receive  feedback  on 
concepts  for  using  smart  sensing  technologies  fused  with  flightline  information  to 
provide  work  and  decision  making  support  to  Expeditors  and  ProSupers  in  order  to 
enhance  resource  utilization  and  maintenance  decisions  for  sortie  production.  The 
purpose  of  SSLC2  is  to  fuse  data  from  smart  sensors,  such  as  RFID/RTLS,  with  current 
flightline  information  and  to  provide  an  interactive  tool  that  presents  decision  quality 
information  to  support  work  performed  by  flightline  personnel.  As  described  in  Section 
3.0,  an  interactive  simulation  test  bed  was  developed  to  allow  maintenance  personnel  to 
interact  with  SSLC2  to  monitor  maintenance  and  resources.  The  simulation  was  designed 
to  demonstrate  different  roles,  including  Expeditor,  ProSuper,  and  MOC.  The  results  of 
this  demonstration  provided  feedback  needed  to  continue  the  development  of 
requirements  for  this  type  of  tool,  and  provided  feedback  related  to  how  the  data  should 
be  fused  and  presented  to  support  maintenance  tasks  and  decisions. 

Two  demonstrations  were  conducted.  One  demonstration  was  an  interactive 
simulation  with  simulated  data  allowing  Expeditors  to  serve  as  subjects  and  interact  with 
the  SSEC2  system  so  they  could  provide  feedback.  The  second  was  a  demonstration  of 
live  tags  placed  on  the  flightline.  Both  demonstrations  took  place  at  the  same  time  at  the 
180th  EW  Toledo  ANG  at  Toledo  Express  Airport,  Swanton,  OH. 

4.2  INTERACTIVE  SIMULATION 
4.2.1  Procedure 

Data  related  to  the  flightline  at  the  Toledo  ANG  were  obtained  which  included 
examples  of  14  unscheduled  maintenance  discrepancies,  7  scheduled  maintenance  tasks, 
personnel,  equipment,  facilities,  and  a  map  of  the  airbase.  These  data  were  used  to  create 
a  simulation  using  the  test  bed.  The  simulation  included  information  related  to  17  aircraft 
with  sorties  and  maintenance  for  a  one  week  (5  day)  period.  The  simulation  begins  at  the 
start  of  the  day  on  Monday.  The  simulation  was  divided  into  four  different  time  periods. 
Time  period  1  began  at  the  start  of  the  day  and  was  used  to  train  subjects  on  how  to 
interact  with  the  SSEC2  software.  The  training  session  took  approximately  30  minutes. 
Eollowing  the  training  session,  three  additional  sessions  took  place,  each  for  a  period  of 
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approximately  10  minutes.  Time  moves  forward  at  the  beginning  of  each  10  minute 
session  and  unscheduled  maintenance  problems  and  other  resource  issues  occur.  The 
subjects  interacted  with  the  system  using  a  tablet  PC  with  a  mouse  and  keyboard  to  solve 
problems,  collaborate  with  other  personnel,  assign  resources,  make  maintenance  and 
fix/swap  decisions,  and  oversee  scheduled  and  unscheduled  maintenance.  They  received 
radio  calls  from  an  experimenter  acting  as  other  flightline  personnel  to  report  issues. 
Table  5  summarizes  demonstration  elements  and  events.  Simulated  data  also  included 
RFID  tags  that  provided  location  and  status  of  resources  (personnel  and  equipment). 


Table  5.  Example  of  demonstration  elements  and  simulated  events 


Session  1:  Training  30  Minutes 

What  is  Being  Demonstrated 

Events 

How  to  view  flying  schedule 

Change  personnel  assigned  to  a  mx  task 

Review  of  maintenance  tasks  (scheduled  and 
unscheduled) 

Receive  a  general  message 

Descriptions  of  personnel  views  (make  a 
personnel  change) 

Description  of  equipment  view 

Location  and  status  information 

Notifications  and  requests  (collaboration) 

Receive  request  for  a  person 

Sending  messages  (manual) 

Making  changes  to  a  task  (edit  subtask) 

Understanding  how  to  swap 

Session  2:  10  Minutes 

What  is  Being  Demonstrated 

Events 

Change  in  A/C  status  on  schedule  view. 

A/C  2098  on  morning  schedule  changes  status 
to  NMC  Nose  Wheel  problem 

Swap 

Expeditor  notified  that  a/c  2151  is  crew  ready 

Milestone  markers  on  schedule  view. 

Expeditor  needs  to  make  a  swap 

Respond  to  request  for  resource. 

Expeditor  reviews  nose  wheel  mx  screens 

Receives  approval  from  ProSuperfor  swap 

Visualization  of  conflict. 

Expeditor  responds  to  a  request  for  a  resource 
from  Expeditor  2. 

Expeditor  reviews  conflict  and  reassigns 
resource  to  remove  conflict. 

Session  3:  10  Minutes 

What  is  Being  Demonstrated 

Events 

Current  time  on  tasks  in  process,  marking 
complete 

Notification  of  landing  (mark  landing  time). 
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Visualization  of  overdue  subtask 

Completion  of  a  subtask,  alert  sent  that  time  is 
past  due,  Expeditor  must  mark  complete. 

Request  a  resource  not  owned 

Extend  task  time  on  nose  wheel  problem, 
cause  ETIC  to  bust. 

View  landing  times 

Need  to  assign  a  new  resources(s)  to  different 
tasks 

Alerts 

ProSuper  assigns  a  resource  causing  a 
conflict  that  expeditor  must  solve. 

New  maintenance  tasks  have  appeared 

Session  4:  10  Minutes 


What  is  Being  Demonstrated 

Events 

SWAP  specified  by  ProSuper 

Aircraft  are  in  process  of  being  turned  for 
afternoon  flights 

New  maintenance  tasks  have  appeared 

AC  2132  on  afternoon  schedule  has 
unscheduled  mx  problem 

Marking  Tasks  complete 

ProSuper  initiates  swap  with  a  spare  that  has 
different  configuration  (automatic  and  manual 
messages  sent). 

Assigning  resources 

2085  came  back  from  am  flight  broke. 

Dealing  with  conflict 

Issues  with  resources  occur  on  tasks. 

Changing  an  ETIC 

Problem  on  one  mx  task  requires  an  extension 
of  ETIC.  Expeditor  requests  ETIC  change. 

During  the  simulation,  subjects  were  asked  to  verbalize  what  they  were  thinking, 
what  action  they  were  taking,  and  why.  Also,  during  each  10  minute  session  the 
experimenter  asked  subjects  3  to  4  questions  to  measure  their  situation  awareness.  The 
experimenter  then  timed  how  long  it  took  to  answer  the  question  using  a  watch  with  a 
second  hand.  This  technique  is  known  as  the  Situation  Present  Assessment  Method 
(SPAM)  [2]. 

During  the  simulation  the  subject  was  responsible  for  the  maintenance  activity  of 
eight  (8)  aircraft.  An  experimenter  was  logged  on  as  a  ProSuper  in  order  to  make 
changes  or  create  events  during  the  simulation  and  to  send  and  respond  to  messages  from 
the  subject.  Events  could  be  initiated  through  the  software  or  via  the  radio. 

At  the  end  of  the  session,  the  subjects  were  asked  to  fdl  out  a  subjective  electronic 
questionnaire  on  a  separate  laptop  computer  (See  Section  4.2.2) 
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4.2.2  Questionnaire 

The  questionnaire  was  designed  to  ask  questions  related  to  seven  (7)  eategories  of 
information  and  to  obtain  some  information  related  to  the  three  Key  Performanee 
Parameters  (KPPs).  The  list  below  indieates  the  type  of  question  and  the  number  of 
questions  in  that  eategory. 

•  Integration  of  the  information  (sensors  with  flightline  maintenanee 
information)  (INF)  -  3 

•  Visualizations  (VIS)  -  5 

•  Situation  Awareness  (SA)  -  6 

•  Resouree  alloeation  (RA)  -  I 

•  Resouree  visibility  (RV)  -  1 

•  Deeision  support  (DS)  -  4 

•  Collaboration  (COL)-  2 

•  Key  Performanee  Parameters  1,  2  and  3.  (One  KPP  was  also  eategorized 
under  DS.) 

Subjeets  were  asked  to  rate  the  effeetiveness  of  SSLC2  on  the  six-point  rating  seale 
shown  below: 

1  -  Not  at  all  effeetive 

2  -  Not  Effeetive 

3  -  Somewhat  Not  Effeetive 

4  -  Somewhat  Effeetive 

5  -  Effeetive 

6  -  Extremely  Effeetive 

In  addition  to  the  rating  questions,  demographie  information  was  eolleeted  related  to 
subjeet’s  maintenanee  experienee.  Two  open-ended  questions  were  asked  at  the  end  of 
the  questionnaire  to  obtain  suggestions  related  to  additional  information  that  should  be 
ineluded  and  suggestions  for  additional  sensed  information. 

4.2.3  Equipment 

The  interaetive  simulation  ineluded  the  equipment  listed  in  Table  6. 


Table  6.  Equipment  and  purpose  for  interactive  simulation 


Equipment 

Purpose 

SSLC2  Server  Dell  Dimension  4700  Desktop 
Computer 

SSLC2  simulation  software 

WhereNet  Server  on  a  Dell  PowerEdge  4005C 
Desktop  Computer 

WhereNet  used  to  simulate  the  RFID  tags 
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Two  Toshiba  Protege  3500  Tablet  PC 
computers 

One  used  by  subjects  to  interact  with  SSLC2, 
one  used  by  experimenter  acting  as  ProSuper 
to  respond  to  subject. 

Dell  Inspiron  8500  Laptop  computer 

Data  collection  to  record  verbal  responses. 

Gateway  600  Laptop  computer 

Electronic  questionnaire 

Radios 

To  allow  communication  from  experimenter 
acting  as  ProSuper  to  subject  (Expeditor) 

Passive  monitor 

To  allow  experimenter  acting  as  ProSuper  in 
separate  room  to  hear  what  was  taking  place  in 
room  with  the  subject. 

4.2.4  SSLC2  Team  Members 

The  simulation  required  four  SSLC2  team  members.  One  experimenter  sat  with  the 
subject,  trained  them  and  asked  the  situation  awareness  questions.  An  experimenter 
acting  as  a  data  recorder  also  sat  in  the  room  with  the  subject  and  recorded  the  verbal 
protocol  on  a  laptop.  A  third  experimenter  was  located  in  the  next  room  and  acted  as  the 
ProSuper  and  other  personnel  on  the  flightline  as  needed  if  the  subject  called  someone. 
This  experimenter  also  initiated  events  by  using  the  SSLC2  software.  A  fourth  person 
controlled  the  SSLC2  server  to  update  the  data  at  the  beginning  of  each  session. 

4.2.5  Subjects 

Five  subjects  from  Toledo  ANG  served  as  subjects.  The  average  number  of  years  they 
have  worked  in  flightline  maintenance  was  22.4  with  a  minimum  of  16  and  a  maximum 
of  30.  Four  of  the  subjects  were  experienced  as  Expeditors  with  an  average  number  of 
years  of  4.35.  Two  subjects  had  experience  as  ProSupers  with  an  average  number  of 
years  of  4.5;  one  subject  was  a  maintenance  controller.  All  subjects  were  male. 


4.3  LIVE  TAG  DEMONSTRATION 

The  purpose  of  this  demonstration  was  to  illustrate  how  live  sensors  and  flightline 
information  are  integrated  to  support  flightline  maintenance  and  to  discuss  any  questions 
that  personnel  on  the  flightline  might  have.  The  simulation  created  for  the  interactive 
demonstration  was  included  to  show  various  aspects  of  the  interface.  Additionally  49 
RFID  tags  were  placed  on  Toledo  ANG  resources.  Personnel  who  had  time  to  visit  the 
demonstration  were  asked  to  fill  out  a  short  questionnaire  if  they  had  time.  Only  four 
personnel  were  able  to  fill  out  the  questionnaire.  The  demonstration  took  place  at  the 
same  time  as  the  interactive  simulation  in  a  separate  room. 
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4.3.1  Equipment 

Table  7  summarizes  the  equipment  used  for  this  demonstration. 


Table  7.  Equipment  and  purpose  for  Live  Tag  demonstration 


Equipment 

Purpose 

SSLC2  Server  Dell  Dimension  4700  Desktop 
Computer 

SSLC2  simulation  software 

WhereNet  Server  on  a  PowerEdge  4005C 
Desktop  Computer 

WhereNet  used  to  simulate  the  RFID  tags 
(Same  server  used  for  interactive  demo) 

Collects  data  from  tags. 

Two  Toshiba  Protege  3500  Tablet  PC 

Computers 

Used  demonstrate  Expeditor  and  ProSuper 
roles  and  visualizations. 

Whereports  and  location  sensors  * 

To  pick  up  sensor  signals 

49  RFID  Tags 

Attached  to  resources  across  the  flightline 

Radios 

To  allow  communication  to  flightline  personnel 
to  move  resources 

InFocus  LP640  LCD  projector 

Project  images  from  tablet  pc  on  wall  to  allow 
multiple  people  to  view  the  demonstration 

5  Results  and  Discussion 

This  seetion  deseribes  the  questionnaire  results  from  the  interaetive  simulation  and 
user  eomments,  eomfort  ratings,  situation  awareness  probes,  questionnaire  results  from 
live  tag  demonstration,  followed  by  a  discussion  of  the  study  results  and  a  summary  of 
how  Spiral  Three  Objectives  were  met. 

5.1  INTERACTIVE  SIMULATION  RESULTS 

5.1.1  Questionnaire  Results 

As  indicated  previously,  the  questions  were  categorized  into  seven  areas.  Results  of 
mean  rating  effectiveness  are  presented  by  categories  below.  Following  each  graph  are 
the  subjects  comments  related  to  the  questions  shown  in  the  graphs. 

5.1 .1.1  Integration  of  Information  (INF) 

Figure  10  presents  the  mean  rating  of  effectiveness  for  three  questions  related  to 
integration  of  information.  Subjects  indicated  that  the  integration  of  data  and  sensor 
information  and  information  detail  was  effective.  Information  organization  nearly 
reached  the  average  effective  (X  =  4.6).  Comments  indicate  that  the  inability  to 
demonstrate  zoom  affected  their  ratings,  although  it  was  explained  to  subjects  that 
zooming  is  a  requirement  and  would  be  available  in  an  actual  system. 
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Integration  of  Information  Questions 


Figure  10.  Mean  effectiveness  rating  for  questions  related  to  integration  of  information 

Subject  Comments  by  Question; 

The  integration  of  information  from  sensors  (status,  location)  with  flightline 
information  (i.e.  flight  schedules,  maintenance,  resources)  is: 

•  “Getting  Familiar  with  where  to  go  is  key.” 

•  “From  a  maintenance  management  perspective,  ifs  nice  to  "see"  where  your 
people  and  resources  are.” 

The  way  in  which  the  information  is  organized  in  SSLC2  is: 

•  “I  like  the  amount  of  information  available.  Again,  getting  familiar  is  key.” 

•  “Need  to  have  the  ability  to  sort  work  sections” 

•  “Tasks  in  the  screens  were  very  busy,  probably  will  ‘be’  fixed  when  you  can 
zoom  the  screens  in  and  out” 

•  “Usable.  Ifs  good  that  I  could  change  between  the  different  screens  (large  to 
small,  small  to  large).” 

The  level  of  information  detail  presented  is:  “Very  good.” 

•  “Very  good.  I  was  able  to  find  people  assigned  to  tasks  and  ensure  they 
weren't  assigned  to  two  tasks  at  the  same  time.” 
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5.1. 1.2  Visualization 

Figure  11  presents  the  mean  rating  effeetiveness  for  questions  related  to  the 
visualizations.  In  general,  the  subjects  found  the  visualizations  to  be  effective.  The 
rating  for  overall  effectiveness  is  close  to  5.0.  Comments  related  to  the  flying  schedule 
indicated  that  ratings  were  affected  by  the  lack  of  zooming  capability  which  was  an 
expected  outcome.  For  alerts  and  notifications,  there  was  no  audible  alarm  to  direct  their 
attention  so  subjects  did  not  always  see  the  messages. 
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Effective 
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Not  Effective 


Not 
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Not  at  ail 
Effective 


Mean  Rating  Effectiveness 
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Figure  11.  Mean  effectiveness  rating  for  visualization  questions 
Subject  Comments  by  Question; 

The  visualization  of  information  in  the  flying  schedule  is: 

•  “Good” 

•  “Need  to  be  able  to  see  more  at  one  time.  Should  be  fixed  with  the  zoom.” 

•  “Again,  would  have  liked  the  zoom  feature,  got  lost  in  the  flying  schedule 
window  by  scrolling  forward  or  back.” 

•  “But,  1  should  be  able  to  view  all  of  the  flyers  at  once,  without  scrolling 
down.” 
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The  visualization  of  unscheduled  and  scheduled  maintenance  task  list  is: 

•  “I  thought  this  was  an  area  where  you  have  a  lot  of  information  available.” 

•  “Would've  liked  to  see  more  of  the  upcoming  hourly/calendar  inspections  or 
maintenance  on  the  Maintenance  page. 

The  visualization  of  unscheduled  maintenance  sub  tasks  and  resources  required  is: 

•  “I  thought  this  was  an  area  where  you  have  a  lot  of  information  available.” 

The  visualization  of  alerts  and  notifications  is: 

•  “It  took  me  a  while  to  get  used  to  looking  for  the  alerts  and  responding  to 
them.” 

•  “Would  rather  hear  a  radio  call.  As  an  expediter  you  need  to  drive  and  look  at 
what’s  going  on.  If  it  is  a  general  message  then  it  could  be  sent  with  maybe  a 
sound  notification  to  let  you  know  you  have  a  message.” 

•  “The  alerts  and  notifications  should  have  an  audible  sound  to  alert  me  to  look 
at  the  bottom.” 

Subject  comments  related  to  Overall  effectiveness: 

•  “I  enjoyed  interacting  with  this  system  and  don't  think  it  would  take  much  to 
operate.” 

5.1 .1 .3  Situation  Awareness 

Figure  12  presents  results  for  questions  related  to  situation  awareness.  In  general 
subjects  indicated  the  system  was  effective  for  situation  awareness,  but  closer  to 
“somewhat  effective”  for  alerts  and  notifications.  This  was  related  to  the  fact  that  the 
system  did  not  provide  an  audible  alert  to  indicate  a  message  was  presented.  Users  had  to 
check  the  messages  which  were  shown  at  the  bottom  of  the  interface. 
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Mx  Tasks  Resource  status  Facility  Overall  flight-  Alerts  and  Prediction  Mx 

and  location  Availability  line  mx  Notifications  tasks/resources 

Situation  Awareness  Questions 

Figure  12.  Mean  effectiveness  rating  for  situation  awareness  questions 
Subject  Comments  by  Question; 

For  providing  situation  awareness  of  maintenance  tasks  and  subtasks  SSLC2  is: 

•  “I  had  the  most  problems  with.  I  continually  forgot  to  click  on  the  subtasks.” 

•  “It's  ideal  for  seeing  the  flow  of  maintenance  actions  required” 

For  providing  situation  awareness  of  resource  (personnel,  equipment,  and  facilities) 
status  and  locations  SSLC2  is: 


•  “This  was  good  and  the  amount  of  resources  loaded  into  the  SSLC2  is 
effective.” 

•  “Knowing  whafs  available  at  any  given  time  is  excellent.” 

For  providing  situation  awareness  of  facility  availability  SSLC2  is: 

•  “I  only  used  this  once  but  was  very  easy  to  use.” 

For  providing  situation  awareness  of  the  overall  flightline  maintenance  SSLC2  is: 

•  “The  program  makes  it  easy  and  fast  to  find  the  overall  awareness  of  flight 
line  issues.” 


For  providing  situation  awareness  of  maintenance  alerts  and  notifications  SSLC2  is: 
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•  “I  didn't  stay  on  top  of  the  alerts  but  they  are  easy  to  see.” 

•  “Need  audible  alerts” 

For  providing  information  to  allow  me  to  prediet  how  changes  may  affect 
maintenance  tasks  and  resource  use  SSLC2  is: 

•  “I  didn't  have  much  interaction  with  this.” 

•  “When  I  extend  a  maintenance  event,  the  following  maintenance  events  (if 
any)  should  automatically  be  extended  by  the  amount  of  time  that  I  extended 
the  first  event.” 

5.1 .1.4  Decision  Support 

Figure  13  presents  results  for  questions  related  to  decision  support.  Decision  support 
was  found  to  be  effective. 
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Decision  Making  Questions 

Figure  13.  Mean  effectiveness  rating  for  decision  making  questions 
Subject  Comments  by  Question: 

For  supporting  my  resource  allocation  decisions  SSLC2  is: 

•  I  can't  stress  enough  that  getting  familiar  with  the  system  is  key. 

For  coordinating  resources  to  meet  multiple  objectives,  SSLC2  is: 

•  “Once  I  got  used  to  where  the  resources  were  located,  this  was  effective” 
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•  “Being  able  to  assign  resourees  to  a  speeifie  task  is  very  useful” 

For  supporting  my  deeision  making  related  to  aireraft  maintenanee  tasks  SSLC2  is: 

•  “Good” 

•  “It  is  very  effeetive  to  see  all  personnel  that  are  available  to  work  aireraft 
maintenanee” 

For  managing  information  needed  during  deeision  making,  SSLC2  is: 

•  “The  system  gave  me  all  the  tools  I  needed  and  in  the  short  amount  of  time  I 
worked  with  it,  it  was  easy  to  use.” 

5.1 .1 .5  Collaboration,  Resource  Visibility,  Resource  Allocation 

Figure  14  shows  the  mean  effeetiveness  rating  for  the  three  eategories  of  questions 

related  to  eollaboration,  resouree  visibility,  and  resouree  alloeation.  Enhaneing 
eollaboration  among  flightline  personnel  was  rated  between  “somewhat  effeetive”  and 
“effeetive”.  One  eomment  indieated  that  it  would  depend  on  whether  people  are  working 
on  tasks  they  are  assigned.  No  one  diseussed  the  eoneept  that  the  system  would 
automatieally  notify  other  personnel  of  a  request.  They  seem  to  foeus  on  eollaboration 
through  the  use  of  messages  and  as  indieated  in  previous  eomments,  no  audible  sound 
was  given  when  a  message  appeared. 
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Subject  Comments  by  Question; 

For  providing  a  shared  vision  of  what  is  occurring  on  the  flightline,  SSLC2  is: 

•  “With  the  SSLC2  available  to  all  supervisors,  this  makes  visual  mgt  easy.” 

•  “Didn't  care  for  having  to  scroll  through  the  aircraft,  would  prefer  being  able 
to  see  all  of  the  jets  at  once.” 

For  enhancing  collaboration  among  flightline  personnel  SSLC2  is: 

•  “Effective  as  long  as  personnel  are  doing  what  they  are  assigned  to  do.” 

For  improving  resource  allocation  and  utilization  (above  current  levels),  SSLC2  is: 

•  “I  feel  this  can  improve  resource  allocation.” 

For  supporting  resource  visibility,  SSLC2  is: 

•  No  subjects  provided  comments 
5.1 .1.6  Key  Performance  Parameters 

Figure  15  provides  mean  effectiveness  ratings  for  three  questions  that  were  related  to 
key  performance  parameters.  KPPl  and  KPP2  questions  where  rated  just  below 


Providing  Enhancing  Improving  resource  Supporting  Resource 

Shared  Collaboration  allocation  and  Utilization  Visibilty 


Collaboration,  Resource  Allocation  and  Visibility 

14.  Mean  effectiveness  rating  for  collaboration,  resource  allocation  and  resource  visibility 

questions 
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Effective.  KPP3,  related  to  reduction  in  time  compared  to  baseline,  was  rated  between 
Somewhat  Effective  and  Effective.  Because  this  is  a  simulated  demonstration  it  is 
difficult  for  users  to  really  know  how  it  may  affect  their  performance  and  time.  An  in- 
depth  field  study  where  users  can  work  with  the  software  in  their  current  environment  is 
needed  to  truly  evaluate  SSEC2  effectiveness. 
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Figure  15.  Mean  effectiveness  rating  for  questions  related  to  key  performance  parameters 

Subject  Comments  by  Question: 

Real  time  status  information  provided  by  SSEC2  as  an  improvement  over  current 
practice  is: 

•  “I  like  that  this  is  up-dated  so  quickly.” 

•  “I  can  see  where  a  finished  product  would  be  a  significant  advantage  but  in 
my  limited  time  in  dealing  with  the  software,  I  think  the  system  needs  to  be  a 
little  more  user  friendly  to  become  a  true  asset  over  current  practices.” 

•  “Knowing  where  equipment  and  resources  are  in  real  time  would  be  very 
useful.” 

Eor  managing  information  needed  during  decision  making,  SSEC2  is: 
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•  “The  system  gave  me  all  the  tools  I  needed  and  in  the  short  amount  of  time  I 
worked  with  it,  it  was  easy  top  use.” 

Reduetion  in  time  to  identify  and  make  deeisions  with  SSLC2  real  time  information 
(eompared  to  eurrent  praetiee)  is: 

•  “Not  sure  if  is  an  improvement  to  ealling  a  shop  direetly  and  requesting 
someone  or  something.” 

•  “Providing  the  information  is  updated  in  real  time,  it  would  be  very  useful  in 
making  deeisions.” 

•  “See  above  eomments”  (This  appears  to  be  referring  to  the  following  previous 
comment:  “I  can  see  where  a  finished  product  would  be  a  significant 
advantage  but  in  my  limited  time  in  dealing  with  the  software,  I  think  the 
system  needs  to  be  a  little  more  user  friendly  to  become  a  true  asset  over 
current  practices.”) 

5.1 .1.7  Open-ended  questions 

Two  open-ended  questions  were  asked  at  the  end  of  the  questionnaire  to  obtain 
suggestions  related  to  additional  information  that  should  be  included  and  suggestions  for 
additional  sensed  information.  The  comments  for  these  questions  are  listed  below. 

Question  1 :  What  additional  information  would  you  like  to  see  included  in  this  type 
of  technology? 

•  “I  don't  have  anything  to  offer  here.” 

•  “Weekly  view  for  scheduled  maintenance.  Weekly/Monthly  view  of  the  flying 
plan.” 

•  “Nothing  at  this  time” 

•  “Maybe  a  main  status  screen  including  all  aircraft,  with  extremely  general 
info,  with  links  to  the  maintenance  tab  and  flying  schedule.” 

•  “Keep  maintenance  functions,  maintenance  functions;  keep  flying  functions, 
flying  functions.  Changing  the  maintenance  ETIC  on  the  flying  board  is  an 
example  of  this.  ETICs  should  come  from  the  maintenance  tab.  My  first 
impression  of  the  program  is  that  it  looks  very  labor  intensive.  Keep  it  simple, 
point  and  click,  dropdown  menus,  etc.” 

Question  2:  What  additional  sensed  information  would  you  like  to  see  included  in  this 
type  of  technology?  (i.e.,  additional  types  of  automatic  sensors). 

•  “Same”  (referring  to:  “I  don't  have  anything  to  offer  here.”) 

•  “Not  sure  at  this  time” 

•  “None  at  this  time” 

•  “Nothing  additional  is  coming  to  mind  at  this  time.” 
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5.1.2  Comfort  Ratings 

After  training  and  at  the  end  of  each  10  minute  simulation  session,  subjects  were 
asked  on  a  scale  of  1-10,  with  10  being  the  most  comfortable,  how  comfortable  they  felt 
with  the  interface.  Table  8  lists  the  results  for  these  ratings.  In  general,  subjects  indicated 
that  the  interface  was  not  difficult  to  use,  but  it  would  take  time  to  become  familiar  with 
where  all  the  data  were  located.  These  ratings  are  very  good  considering  that  the  users 
were  only  able  to  interact  with  the  system  over  a  few  hours.  This  indicates  that  overall  the 
system  was  easy  to  use  and  learn.  A  usability  test  was  not  conducted  because  of  time  and 
cost  limitations. 


Table  8.  Comfort  ratings  across  subjects 


Subject 

Number 

Training  Session 

Session  1 

Session  2 

Session  3 

1 

8 

8 

8 

8 

2 

7 

6.5 

6.5 

6 

3 

6.5 

7 

7 

7 

4 

7 

8.5 

8.5 

8.5 

5 

6 

7 

6 

Missing  data 

Mean 

6.9 

7.4 

7.2 

7.4  (n=4) 

5.1.3  Situation  Awareness  Probe  Results 

During  the  simulation  the  experimenter  would  ask  the  subject  a  situation  awareness 

probe  question.  The  time  to  answer  the  question  was  recorded.  In  some  cases  the 
questions  were  not  asked  or  the  timing  data  were  not  accurately  collected.  However,  the 
technique  did  provide  some  limited  insight  into  their  ability  to  know  where  to  find  data. 
Table  9  lists  the  probe  questions,  the  mean  time  to  answer  and  minimum  and  maximum 
times.  In  general  subjects  were  able  to  find  the  data  within  a  minute.  One  subject  had 
difficulty  finding  the  next  phase  inspection  information  taking  at  total  of  240  seconds, 
whereas  all  other  subjects  found  this  data  in  30  seconds  or  less.  One  subject  also  had 
difficulty  determining  the  number  of  avionics  specialist’s  that  were  currently  assigned  to 
aircraft  requiring  help  but  timing  data  were  not  correctly  recorded. 
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Table  9.  Average  time  to  answer  probe  questions 


Session  1 

N 

Mean 

(seconds) 

Maximum 

Minimum 

Where  is  Crew  Chief  “name” 

4 

29 

60 

15 

Where  is  “name”  right  now 

2 

37 

25 

25 

What  equipment  would  be  used  to  do  OPS  check 
on  2098? 

3 

43 

60 

30 

Session  2 

What  certification  does  (person’s  name)  have? 

5 

38 

50 

20 

How  many  EE  are  currently  assigned? 

5 

48 

60 

30 

When  is  next  phase  inspection  for  a/c  “X” 

5 

66 

240 

12 

Session  3 

What  Avionics  Specialist  is  assigned  to  a/c  “X”? 

4 

42 

75 

15 

Given  the  maintenance  issues  you  see  how  will 
this  affect  your  schedule  tomorrow? 

4 

61 

110 

13 

Total  Average 

46 

5.2  LIVE  DEMONSTRATION 

A  questionnaire  with  1 1  questions  was  available  for  personnel  who  participated  in  the 
live  tag  demonstration.  Four  people  turned  in  questionnaires.  The  results  are  listed  in 
Table  10,  with  the  number  for  each  question  indicating  the  frequency  of  the  response. 
The  small  sample  size  makes  it  difficult  to  draw  any  specific  conclusions;  however,  the 
numbers  are  similar  to  the  questionnaire  results  for  the  five  subjects  who  interacted  with 
the  system  during  the  interactive  simulation.  Overall,  the  subjective  impressions  are  very 
positive. 

The  two  open-ended  questions  that  were  included  in  the  interactive  simulation  were 
also  included  on  this  survey.  The  comments  for  these  questions  are  listed  below. 

Question  1 :  What  additional  information  would  you  like  to  see  included  in  this  type 
of  technology? 

•  Include  Sender  in  messaging  option 

•  Tool  Software  Integration 

Question  2:  What  additional  sensed  information  would  you  like  to  see  included  in  this 
type  of  technology?  (i.e.,  additional  types  of  automatic  sensors). 

•  No  comments  provided 
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Table  10.  Questionnaire  results  for  the  live  demonstration 


CODE 

Statements 

Not  at  all 
Effective 

1 

Not 

Effective 

2 

Somewhat 

Not 

Effective 

3 

Somewhat 

Effective 

4 

Effective 

5 

Extremely 

Effective 

6 

INF 

The  integration  of  information  from 
sensors  (status,  location)  with  flightline 
information  (i.e.,  flight  schedules, 
maintenance,  resource)  is: 

4 

DS,  KPP2 

For  managing  information  needed  during 
decision  making,  SSLC2  is: 

2 

1 

1 

COL 

For  enhancing  collaboration  among 
flightline  personnel  SSLC2  is 

1 

3 

RA 

For  improving  resource  allocation  and 
utilization  (above  current  levels),  SSCL2 
is 

4 

RV 

For  supporting  resource  visibility,  SSLC2 
is: 

3 

1 

KPP  1 

Real  time  status  information  provided  by 
SSLC2  as  an  improvement  over  current 
practice  is: 

2 

1 

1 

KPP3 

Reduction  in  time  to  identify  and  make 
decisions  with  SSLC2  real  time 
information  (compared  to  current  practice) 
is: 

1 

3 

VIS 

The  visualizations  of  resource  status  and 
locations  are: 

2 

2 

Overall,  SSLC2  appears  to  be: 

4 
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5.3  DISCUSSION 


5.3.1  User  Performance 

The  questionnaire  results,  eomfort  ratings,  and  timing  results  for  situation  awareness 
probe  questions  indieate  that  overall  subjeets  found  the  SSLC2  system  to  be  effeetive  and 
useful.  Subjeets  mentioned  that  their  eomfort  level  was  related  primarily  to  the  faet  that 
they  needed  time  to  praetiee  with  the  system  given  the  large  amount  of  data  available  to 
them.  Ratings  were  sometimes  affected  by  the  lack  of  zoom  and  filtering  functions.  For 
example,  comments  include  the  need  to  view  weekly  and  monthly  schedules.  With 
zoom,  they  would  be  able  to  see  the  schedule  at  any  level,  but  this  was  not  demonstrated. 
There  was  also  a  general  feeling  that  messages  and  alerts  would  require  audio  tones, 
however,  this  must  be  carefully  evaluated  based  of  how  many  they  may  receive.  Too 
many  messages  and  alerts  could  be  overwhelming. 

It  is  interesting  to  note  that  subjects  did  not  indicate  that  any  information  was  missing 
and  subjects  only  occasionally  thought  data  should  be  located  in  a  different  place. 
Subjects  were  able  to  find  most  information  quickly  according  to  the  situation  awareness 
probe  results.  In  some  cases  the  experimenter  needed  to  help  them  find  information, 
however,  most  subjects  indicated  that  they  just  needed  more  time  to  become  familiar  with 
the  interface. 

One  subject  requested  information  related  to  personnel  by  work  centers.  This  is  due  to 
the  fact  that  filtering  was  not  available  during  the  demonstration,  but  there  is  a 
requirement  to  sort  or  filter  personnel  by  work  centers. 

The  use  of  verbal  protocol  during  the  simulation  allowed  us  to  understand  how  users 
were  proceeding  through  the  interface  and  to  determine  issues  they  had.  From  the  verbal 
protocol  and  questionnaire  results  we  found  that  the  interface  allowed  the  subjects  to 
quickly  see  when  things  on  the  flightline  were  beginning  to  go  wrong  (i.e.  maintenance 
problems  started  occurring).  If  they  saw  a  problem  they  would  determine  if  that  aircraft 
was  flying  that  day.  They  would  concentrate  on  aircraft  that  were  flying  first.  Checking 
for  this  information  required  moving  to  the  flying  schedule.  To  reduce  the  need  to  move 
too  often  between  screens,  an  indication  of  the  next  scheduled  flight  time  for  any  broken 
aircraft  could  be  provided  on  the  maintenance  screen. 
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Table  11  indicates  issues  that  occurred  during  the  interactive  simulation  and  some 
possible  solutions  based  on  the  existing  interface. 


Table  11.  Issues  noticed  during  interactive  simulation  and  possible  solutions 


Issue 

Possible  Solutions 

User  looking  for  exceptional  release.  Realized 
later  that  crew  ready  was  exceptional  release. 

Related  to  definitions.  Can  be  solved  with 
limited  training. 

Issue  with  finding  personnel  that  were  in 
conflict.  The  maintenance  subtask  screen 
indicated  a  conflict  but  did  not  indicate 
specifically  personnel  that  were  in  conflict. 
Requires  users  to  look  in  mx  views  and 
personnel  views. 

Conflict  data  should  be  visible  in  multiple 
locations  including  detail  views.  The  detail 
view  could  highlight  specific  tasks  the  resource 
is  assigned  to  that  are  in  conflict. 

Not  enough  information  on  certifications  with 
just  numbers. 

Names  of  certification  required  rather  than  just 
certification  numbers. 

Some  difficulty  in  finding  task  end  times. 

Suggestion  for  including  subtask  time  in  details 
view  when  the  top  bar  of  the  subtask  is 
selected  in  subtask  mx  view. 

Suggestion  for  the  need  for  a  breakdown  of 
information  by  work  centers. 

Subject  suggested  tabs  at  top  for  different  work 
centers  when  looking  at  personnel.  Filtering 
requirement  solves  this  problem. 

Aircraft  ETIC  is  on  flying  schedule.  Subject 
considers  it  maintenance  issue  so  should  be  on 

mx  screens 

Add  ETIC  to  mx  subtask  details  screen. 

Difficulty  performing  swap  from  the  swap  dialog 
box.  Selection  was  difficult  at  times  and  the 
highlighting  of  sorties  was  confusing. 

Consider  redesigning  swap  function.  Drag  and 
drop  functionality  not  included  in  the  test  bed 
would  solve  this  problem. 

Subject  wants  more  information  on  what  tasks 
are  coming  up  in  terms  of  all  scheduled 
inspections. 

This  information  is  in  the  mx  view,  however, 
data  were  only  available  for  one  week  for  the 
demo.  Consider  design  of  screens  that 
provides  quick  look  summary  of  schedule 
maintenance  up  to  one  month  in  the  schedule. 

Subject  looked  for  phase  in  mx  screens. 

Phase  was  in  the  flying  schedule  view.  Can 
easily  also  add  to  details  view  under  mx 
screen. 

Reassigning  personnel  to  tasks  was 
sometimes  confusing. 

Consider  eliminating  the  remove/reassign 
dialog  box  and  use  drag  and  drop  functionality. 
Must  consider  how  to  maintain  useful  conflict 
checking. 

Include  the  ability  to  assign  a  new  spare.  After 
a  swap  is  performed  using  the  spare  the  user 
may  want  to  assign  another  aircraft  to  be  an 
additional  spare.  This  will  enable  a  quick  swap 
decision  to  be  made  if  required  when  a  launch 
is  pending. 

In  addition  to  swap  function  include 
reassignment  functions. 
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Issue 

Possible  Solutions 

Identify  when  an  aircraft  requiring  a  certain 
configuration  for  a  sortie  does  not  currently 
have  that  configuration. 

Determine  best  technique  to  bring  to  the  user’s 
attention  that  a  configuration  difference  exists. 

Ability  to  quickly  see  labels  for  items  on  the  geo 
view. 

Resources  on  the  geo  view  may  be  stacked. 

To  identify  a  specific  item  labels  should  be 
available.  Currently  if  icons  are  stacked,  user 
must  click  on  the  icon  and  a  dialog  box  with  list 
of  equipment  appears.  Non-stacked  icons  do 
not  have  labels.  Techniques  to  consider  are 
labels  appear  when  cursor  hovers,  or  ability  to 
turn  on/off  labels. 

Automatically  extend  entire  task  time  when 
user  edits  the  time  of  a  subtask. 

Automatically  extending  times  when  subtask 
was  changed  was  not  included  in  this  spiral 
based  on  time  constraints.  This  functionality 
should  be  included. 

When  selecting  an  aircraft  on  the  schedule 
view,  the  aircraft  details  are  related  only  to  the 
next  sortie.  Include  the  ability  to  see  details 
related  to  any  scheduled  sorties. 

Consider  allowing  the  user  to  select  the  line 
number  to  show  details  related  to  the  aircraft 
for  that  sortie. 

Refresh  was  slow. 

Need  to  increase  refresh  speed  and  determine 
best  configurations  for  refresh  frequencies. 

5.3.2  Testimonials 

After  completion  of  the  field  demonstration,  SSLC2  team  members  went  back  to  the 
Toledo  ANG  to  debrief  and  obtain  any  additional  comments  they  may  have  after  having 
thought  about  the  system.  During  that  time  the  lead  ProSuper  discussed  his  views  of  the 
SSLC2  system.  He  indicated  to  researchers  that  currently  it  takes  15  minutes  to  collect 
the  information  needed  to  make  a  decision  about  whether  or  not  an  aircraft  will  meet  a 
sortie.  With  this  system  he  stated  that  they  could  make  that  decision  in  seconds.  He  also 
stated  the  system  could  reduce  the  time  to  make  decisions  and  assign  resources  to 
maintenance  tasks  by  up  to  60%. 

5.3.3  Meeting  SSLC2  Spiral  Three  Objectives 

Spiral  Three  had  five  objectives  as  outlined  in  Section  3.0.  This  section  summarizes 

the  work  completed  to  meet  each  objective. 

Objective  1;  Develop  smart  sensors  for  logistics  operations. 

During  Spiral  Three  Alion  Science  and  Technology,  Inc.  began  development  of  smart 
tags  based  on  Spiral  One  field  test  feedback  to  provide  status  information  in  addition  to 
location  information.  They  developed  a  smart  tag  that  provided  two  levels  of  status 
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(available/not  available,  on/off).  Due  to  budgeting  eonstraints  the  tags  were  not  ineluded 
in  the  field  demonstration. 

The  SSLC2  test  bed  simulated  the  use  of  smart  tags  that  provided  information  about 
resouree  status  ineluding  broken,  in  use,  and  available.  During  the  demonstration, 
fiightline  maintainors  were  asked  if  there  was  additional  sensed  information  that  might  be 
useful.  They  did  not  provide  any  speeifie  suggestions.  Reeommendations  for  smart 
sensors  are  discussed  under  6.0  Recommendations. 

Objective  2:  Develop  a  method  for  determining  how  data  should  be  fused  for 
presentation  that  focuses  on  supporting  decisions  and  collaboration. 

When  designing  interactive  visualizations  to  support  human  users,  a  user-centered 
approach  is  required.  For  many  complex  systems  the  amount  of  information  and  the 
number  of  decisions  users  make  during  their  work  can  be  very  large.  Mapping  the 
information  to  the  design  to  support  decisions  is  important,  but  visualization  design  is 
somewhat  of  an  art  form.  During  this  spiral  we  developed  a  rating  technique  to  allow 
SMEs  to  rate  the  importance  of  information  across  decisions  and  performed  a  cluster 
analysis.  The  cluster  analysis  was  very  helpful  for  developing  constraints  on  how  to 
integrate  large  amounts  of  data  for  arrangement  in  the  interface.  This  data  combined  with 
an  understanding  of  how  users  perform  their  work  and  what  information  is  needed  when 
they  are  making  decisions  helped  in  developing  interactive  visualizations  to  support 
users.  While  there  was  limited  testing  possible  in  Spiral  Three  due  to  resource 
constraints,  the  integrated  visualizations  received  more  positive  comments  than  did  the 
Spiral  One  interface.  The  layout  of  data  was  not  the  primary  issue  that  users  mentioned 
during  the  simulation.  The  inability  to  include  the  zoom  and  filtering  functionality  with 
this  complex  interface  affected  user  perceptions  even  though  they  were  told  during 
training  the  interface  would  include  zooming  and  filtering  in  a  future  phase.  This  was  not 
unexpected  as  it  is  difficult  for  the  users  to  conceptualize  the  functionality.  This  is 
another  example  of  how  constraints  can  affect  design. 

To  truly  investigate  the  usefulness  of  cluster  analysis  in  design,  research  should  be 
conducted  to  determine  if  this  technique  results  in  improved  interface  designs.  A  study 
could  be  developed  that  provides  multiple  groups  of  designers  with  information  to  design 
a  user  interface.  All  design  teams  would  receive  the  same  information,  but  some  teams 
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would  also  receive  cluster  analysis  results.  User  interface  testing  on  the  resulting  designs 
would  help  to  establish  the  technique’s  effectiveness. 

Objective  3;  Obtain  user  opinions  of  deployed  sensors  in  the  field. 

A  field  demonstration  was  conducted  to  obtain  user  opinion  and  performance  during  a 
simulated  interactive  scenario  and  during  a  live  tag  demonstration.  The  SSLC2 
successfully  completed  this  objective.  The  results  indicate  that  in  general  SSLC2 
concepts  are  effective. 

Objective  4;  Develop  a  simulation  test  bed  and  evaluate  user  feedback  on  integrated 
data  fusion  decision  support  concepts. 

The  design  concepts  were  programmed  into  a  simulation  test  bed  as  described  in 
Section  3.0  of  this  report.  The  SSLC2  team  successfully  completed  this  objective. 

Objective  5;  Develop  recommendations  for  transition  to  existing  systems. 

The  information  gathered  during  Spiral  Three  and  in  the  previous  Spiral  One  has 
provided  specific  recommendations  outlined  in  Section  6.0  of  this  report.  Additionally, 
the  design  concepts  described  throughout  this  paper  and  Use  Cases  can  be  considered 
design  recommendations.  The  SSLC2  team  successfully  completed  this  objective. 

6  Recommendations 

The  SSCL2  research  conducted  during  Spirals  One  and  Three  has  provided  a  great 
deal  of  input  toward  the  design  of  an  interactive  decision  support  system  that  fuses  sensor 
data  with  flightline  data.  The  results  of  this  research  provide  specific  requirements  for  a 
SSLC2  system.  This  section  provides  additional  recommendations  and  future  research 
needs. 

6.1  SMART  SENSORS 

In  Spiral  One  it  was  recommended  that  smart  sensors  be  developed  that  included 
information  in  addition  to  location.  In  Spiral  Three,  a  smart  tag  was  designed  to  track 
personnel  location  that  could  also  indicate  status.  Also,  smart  sensors  with  status 
information  for  equipment  and  personnel  resources  were  simulated  in  the  field 
demonstration.  Equipment  status  included  broken,  in  use,  and  available.  Personnel 
included  available  or  not  available.  The  users  found  this  information  useful  and  chose 
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personnel  for  tasks  based  on  their  availability.  Comments  indieated  that  users  like  the 
ability  to  easily  find  personnel.  Although  users  were  asked  if  there  was  any  other 
information  they  would  like  sensed,  they  indicated  they  could  not  think  of  any. 
Recommendations  for  additional  information  that  could  be  sensed  and  integrated  include 
the  following; 

•  Supplies  on  the  quick  reference  list; 

•  Equipment  fuel  level; 

•  Liquid  oxygen  level; 

•  Temperature; 

•  Battery  level;  and 

•  Oil  levels. 

The  information  from  sensors  that  can  provide  this  information  could  easily  be 
implemented  into  the  visualization  interface  without  major  interface  changes  by 
providing  the  information  in  the  details  screens.  Additional  sensed  information  could  be 
evaluated  to  determine  how  and  when  they  would  be  used  so  that  it  can  be  integrated  into 
the  system  effectively.  Efficient  use  of  sensors  and  integration  of  the  information  is 
critical  and  the  potential  for  information  overload  is  very  high.  It  is  recommended  that 
information  related  to  critical,  Low  Density/High  Demand  (LD/HD)  items  be  sensed. 

In  addition  to  the  items  listed  above,  smart  sensors  that  can  be  used  to  keep  track  of 
the  aircraft  configuration  would  be  very  useful.  This  recommendation  is  discussed  in 
section  6.6  below.  Configuration  Tracking. 

6.2  USER  INTERFACE 

The  user  interface  concepts  were  developed  by  following  a  user-centered  approach 
where  cognitive  task  analysis  was  used  as  input  to  the  design.  The  use  of  cluster  analysis 
to  help  determine  how  information  should  be  grouped  for  design  in  the  interface  was  a 
very  useful  technique  and  resulted  in  an  interface  that  was  easy  to  use  and  allowed  a  great 
deal  of  information  to  be  easily  available  to  the  user  when  needed. 

As  with  any  project,  there  were  time  and  cost  constraints  which  affected  some  design 
choices  for  the  test  bed.  However,  the  interface  still  allowed  concepts  to  be  presented  to 
users  for  important  feedback.  Eor  future  research  and  development  of  SSLC2,  updates  to 
the  user  interface  are  recommended  and  described  below. 
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6.2.1  Portability  and  Touch  Screen  Interface 

The  SSLC2  system  requires  portability  beeause  users  are  always  moving;  therefore, 

the  interfaee  should  limit  the  need  for  a  great  deal  of  keyboard  entry.  The  tablet  PCs  used 
for  the  field  demonstration  did  not  allow  for  easy  toueh  sereen  tapping  and  a  keyboard 
was  needed  for  input.  A  mouse  was  used  for  input.  The  use  of  tablet  PCs  with 
handwriting  reeognition  and  toueh  sereen  interfaee  is  important.  Newer  systems  are  more 
advaneed  as  is  illustrated  by  new  gaming  teehnology. 

6.2.2  Drag  and  drop 

While  the  overall  eoneept  and  navigation  for  the  interfaee  was  easy  and 
understandable,  too  many  menu  interaetions  and  dialog  boxes  with  keyboard  inputs  were 
needed.  As  reeommended  in  Spiral  One,  the  direct  manipulation  interface  should  include 
drag  and  drop  capabilities  and  users  did  mention  this  during  the  field  test.  For  example, 
swapping  an  aircraft  required  a  dialog  box.  However,  dragging  the  spare  aircraft  or 
aircraft  to  be  swapped  to  the  line  would  be  feasible.  At  that  point  the  system  can  check 
for  conflicts  and  if  there  are  any,  provide  a  dialog  box  that  indicates  a  conflict  and  asks 
users  if  they  would  like  to  continue.  If  this  capability  is  included  the  user  should  also 
have  an  option  to  easily  see  what  aircraft  are  recommended  for  the  swap  and  what  the 
current  configuration  is.  Another  example  is  editing  subtasks  which  required  a  dialog 
box.  The  ability  to  use  the  input  device  and  perform  a  tap  that  is  similar  to  a  right  mouse 
click  could  allow  the  user  to  edit  portions  of  the  subtask  directly  from  the  subtask  box  in 
the  maintenance  screen.  New  concepts  for  manipulating  the  data  should  be  designed, 
explored  and  tested  before  implementation. 

6.2.3  Tailored  Interface 

Users  indicated  they  would  like  to  be  able  to  tailor  the  interface  for  their  use.  For 
example,  they  would  like  to  tailor  the  view  of  the  windows  to  indicate  if  and  when  the 
different  views  (geo  and  detail)  were  available.  They  may  also  like  to  tailor  the  order  of 
the  information  in  the  details  screen  and  be  able  to  turn  on  or  off  information  in  the 
views.  For  example,  some  users  would  like  to  have  the  job  control  number  linked  with 
CAMS  in  the  data,  while  others  do  not.  Future  spirals  should  consider  how  the  interface 
can  be  tailored  and  test  approaches. 
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6.3  TREND  ANALYSIS 

During  Spiral  One  it  was  recommended  that  SSCL2  have  the  capability  of  allowing 
users  to  “replay”  or  review  what  occurred  over  a  specific  time  period  to  enable  them  to 
study  decisions  and  impacts  for  trend  analysis  and  lessons  learned.  This  trend  analysis  is 
a  continued  recommendation  and  requires  study  to  determine  the  best  way  in  which  to 
analyze  and  present  the  data  in  order  for  it  to  be  useful.  Capturing  all  the  history  data  so 
the  user  can  go  back  and  either  replay  the  past  events  or  look  at  the  specific  history  of  a 
specific  resource  would  be  very  valuable  for  trend  analysis  and  resource  allocation 
decisions.  Smart  Systems  could  provide  a  review  of  what  occurred  over  a  previous  time 
period  including  changes  to  schedules,  resources,  and  their  status. 

6.4  FLIGHTLINE  PERFORMANCE  METRICS 

Flightline  performance  metrics  for  establishing  fleet  health  are  an  important  part  of 
flightline  maintenance  and  should  be  included  as  part  of  SSLC2.  Spiral  Three  did  not 
focus  on  this  requirement;  however,  as  indicated  in  the  conceptual  design  the  system 
could  include  performance  metrics  on  a  separate  tab.  While  not  included  in  the  Spiral 
Three  conceptual  design  the  following  information  was  collected  which  should  be 
considered  for  monitoring  and  assessing  fleet  health  for  future  spirals.  This  list  includes 
performance  metrics  currently  defined  in  Air  Force  documents  as  well  as  other  data  that 
could  be  considered  part  of  trend  analysis. 

•  Identify  equipment  resources  trends,  for  example  equipment  that  are  down  for 
specified  periods  of  time  (day,  or  days). 

•  Specify  trends  for  usage  of  facilities  over  past  week,  month.  Specify  when 
facilities  where  not  available  or  in  use  when  needed  (i.e.  conflicts  that  arose). 

•  Specify  NMC  rate  over  past  week,  month. 

•  Specify  number  of  times  CANN  was  necessary. 

•  Specify  number  of  times  aircraft  swap,  delay,  cancel,  late  takeoffs  and 
landings,  aborts  occurred  over  past  week,  month  and  indications  for  causes  for 
these  flying  schedule  deviations. 

•  Show  trend  data  for  reasons  for  flying  schedules  deviations  (lack  of  personnel, 
equipment,  facilities,  supplies,  weather,  crew  delay). 

•  Specify  trends  of  equipment  usage  as  a  function  of  availability,  over  time. 
Indicate  when  thresholds  have  been  reached. 

•  Specify  equipment  failure  rates  over  time  (specific  equipment  and  overall 
equipment). 
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•  Specify  overtime  rates  and  utilization  rates  for  work  centers  over  time. 

•  Identify  personnel  availability  for  current  week  and  month  including  their 
qualifications,  etc. 

•  Specify  the  balance  of  skills  over  time  and  what  it  should  be. 

•  Identify  repeat  discrepancies. 

•  Specify  general  times  to  perform  maintenance  tasks  against  recommended 
maintenance  times.  (May  indicate  training  needs). 

•  Show  trends  for  personnel  availability  over  past  month. 

•  Indicate  specific  aircraft  maintenance  trends  over  time  based  on  discrepancies 
(recurring  malfunction  trends). 

•  Specify  aircraft  subsystems  maintenance  trends  over  time  based  on 
discrepancies  (recurring  malfunction  trends). 

•  Specify  systems  that  are  fixed  on  site  vs.  depot  fix  trends. 

•  Specify  supply  stock  trends. 

•  Flying  schedule  effectiveness:  (Actual,  goal,  trend  by  month). 

•  Flying  schedule  deviations  (month  based  on  reason). 

•  Abort  rate 

•  Mission  Capable  (MC)  rate 

•  Total  Not  Mission  Capable  Rate  For  Maintenance  (TNMCM)  rate 

•  CANN  rate 

•  Break  rate 

•  8  or  12  hr  fix  rate 

•  Repair  cycle 

•  Chargeable  deviations 

•  Engine  running  crew  change 

•  Ground  abort  (discrepancy  after  aircrew  arrival  such  that  crew  can  not 
complete  scheduled  mission). 

•  Maintenance  non-delivery:  sortie  cancel  due  to  maintenance. 

•  Operations  non-delivery:  late  crew  show,  schedule  conflicts  for  instructor  or 
student;  late  return  from  previous  sortie.  Foreign  Object  Damage  (FOD),  etc. 

•  Supply  non-delivery  lack  of  repair  parts,  fuel,  late  delivery  of  service  (fuel, 
oil). 

•  Deletions:  sorties  scheduled  and  not  flown. 

•  Additions:  aircraft  added  to  weekly  schedule  (maintenance  and  operations). 

•  Weather  Additions 

•  Weather  deletions 

•  NMC  aircraft  replacement 

•  Air  abort 
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6.5  COLLABORATION  AND  SHARED  VISION 

Flightline  maintenance  requires  significant  collaboration  among  people  and  work 
shops.  The  ability  for  users  to  collaborate  and  easily  view  the  status  of  the  fhghthne, 
including  maintenance  task  requirements  and  key  resources  and  their  status  is  the  key  to 
increasing  operational  capability  and  sortie  production  and  to  optimizing  the  use  of 
resources.  During  the  field  test  demonstration  users  mentioned  how  they  thought  the 
system  would  be  very  useful  for  other  users  such  as  MOC,  allowing  everyone  to  know 
what  was  going  on.  Spiral  Three  included  collaboration  by  including  multiple  users  in 
the  design  (ProSuper,  MOC,  work  shops),  and  by  including  the  ability  to  send  manual 
messages  and  to  send  and  received  automated  messages  that  affect  maintenance  and 
resource  use.  However,  this  is  just  a  starting  point  for  possible  improvements  in 
collaboration.  The  system  could  assist  collaboration  in  additional  ways.  For  example, 
with  the  inclusion  of  data  analysis  that  supports  analysis  of  trends  and  with  the  data 
collection  on  fhghthne  metrics,  the  system  could  create  reports  for  daily,  weekly  and 
monthly  meetings,  similar  to  Spiral  Two,  VSLRC  [11].  Additionally,  smart  reports  could 
be  generated  at  various  shops  and  Expeditors  and  ProSupers  providing  summaries  of 
resource  and  maintenance  status  at  the  beginning  of  a  shift,  thus  allowing  users  a  quick 
glance.  They  could  then  move  deeper  into  the  system  for  more  specific  information. 

The  ability  to  send  and  receive  messages  via  the  communication  window  is  another 
form  of  collaboration.  The  current  field  test  included  the  ability  to  send  manual  messages 
and  system  generated  messages.  This  function  was  included  for  testing  based  on  findings 
from  Spiral  One  which  indicated  that  some  resources  need  to  be  shared  across  Expeditors 
and  also  across  different  maintenance  groups  when  deployed.  If  a  resource  is  owned  by  a 
different  maintenance  group  the  Expeditor  must  call  to  borrow  the  resource.  SSEC2  lets 
the  user  see  what  equipment  are  available  even  across  maintenance  groups  and  sends 
requests  automatically. 

System  generated  messages  implemented  and  considered  but  not  implemented  are 
listed  in  Table  12. 
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Table  12,  Messages  and  Alerts 


Reason  for  Message  or  Alert 

Implemented 
Spiral  Three? 

1 .  An  Expeditor  has  assigned  personnel  or  equipment,  not  under  his 
control,  to  perform  a  task.  Requesting  a  resource. 

YES 

2.  A  supervisor  (i.e.,  ProSuper,  Maintenance  Supervisor)  changes  a 
resource  that  the  user  (i.e..  Expeditor  of  a  specific  aircraft)  is  in  charge 
of  monitoring. 

YES 

3.  A  supervisor  or  another  user  changed  the  completion  time  on  a 
maintenance  sub-task. 

a.  Threatens  to  affect  aircraft  ETIC 

b.  Threatens  entire  task  completion  time  but  not  ETIC 

c.  Does  not  threaten  entire  task  completion  time. 

YES 

4.  A  sub-task  is  delayed  (i.e.,  has  not  finished  within  specified  time  period) 
and  therefore  threatens  meeting  the  task  completion  time 

a.  Threatens  to  affect  aircraft  ETIC 

b.  Does  not  threaten  aircraft  ETIC. 

YES 

5.  Expeditor  or  ProSuper  changes  the  ETIC  of  an  aircraft. 

a.  The  ProSuper  changes  the  aircraft  ETIC  and  it  does  not  exceed 
crew  ready  time.  (  Notice  sent  to  MOC  and  Expeditor) 

b.  The  ProSuper  changes  the  aircraft  ETIC  and  it  does  exceed  crew 
ready  time.  (Notice  sent  to  MOC  and  Expeditor) 

c.  The  Expeditor  requests  an  ETIC  change  and  it  exceeds  crew  ready. 
(Notice  sent  to  ProSuper) 

d.  The  Expeditor  requests  an  ETIC  change  and  it  does  not  exceed 
crew  ready.  (Notice  sent  to  ProSuper). 

YES 

6.  A  swap  is  initiated. 

a.  A  swap  is  initiated  by  the  ProSuper 

b.  A  swap  is  initiated  by  the  Expeditor 

YES 

7.  A  schedule  change  has  been  issued,  meaning  that  a  2407  has  been 
generated.  Note:  Only  a  MOC  can  approve  a  schedule  change  (AF- 
2407) 

a.  Action  required 

b.  Notification  no  action  required 

NO 

8.  Approval  of  an  AF-2407 

NO 

9.  In-process  inspection  required 

NO 

Additional  possible  messages/alerts  to  consider  include; 

•  Alert  when  a  new  discrepancy  is  reported. 

•  Alert  when  the  aircraft  is  scheduled  to  start  take-off  or  landing  sequence. 

•  Alert  to  highly  dangerous/  hazardous  maintenance  tasks. 

•  Alert  when  certain  resource  levels  are  reaching  a  specified  limited  threshold 
(e.g.  when  resources  are  getting  low  or  are  all  assigned  for  a  specific  time 
period). 

•  Alert  when  resource  is  needed  but  conflict  exists. 
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Alerting  can  be  used  as  decision  support.  For  example,  decision  support  systems  for 
prescribing  medicines  rely  specifically  on  alerts  for  indicating  specific  drug  interactions 
and  information  to  pharmacists  and  physicians.  However,  a  drawback  to  alerting  is  that 
the  users  can  become  overwhelmed  with  too  many  alerts  resulting  in  alert  fatigue. 
Therefore  it  is  important  to  determine  messages  and  alerts  that  should  be  included  in  the 
system  and  also  additional  ways  in  which  to  provide  the  information  to  users. 

Users  in  the  field  study  indicated  that  they  needed  an  audio  alert  to  let  them  know 
when  a  message  arrived.  This  is  possible,  but  needs  to  be  considered  with  the  fact  that 
they  are  often  in  a  high  noise  environment.  Some  alerts  may  need  audio  while  others  of 
lower  importance  might  not  have  audio.  If  an  alert  is  very  serious  the  alert  can  also  be 
displayed  on  top  of  their  working  window. 

6.6  CONFIGURATION  TRACKING 

Configuration  of  the  aircraft  is  very  important  information  for  assigning  aircraft  to 
sorties  and  keeping  track  of  maintenance  subtasks  that  require  re-configuration  or  that 
require  the  downloading  of  weapons  before  maintenance  can  take  place.  SMEs  indicated 
that  tracking  and  control  of  configurations  is  currently  difficult  and  time  consuming.  An 
aircraft’s  configuration  is  currently  identified  on  the  operational  schedule.  The 
configuration  consists  of  information  related  to  Tanks/Pods,  Under-wing  storage. 
Missiles,  Guns,  and  AMD/Fuses.  SSCL2  could  be  used  to  help  users  to  track  and  control 
configurations.  There  are  several  possibilities  for  including  this  in  SSLC2.  One  would  be 
to  include  a  function  that  allowed  the  user  to  input  changes  to  the  configurations  to 
update  the  SSCL2  data.  Another  future  possibility  is  the  use  of  sensors  on  the  aircraft 
equipment.  Thus,  the  configuration  can  be  automatically  sensed  and  provided  to  the 
system.  This  is  feasible  but  will  require  research  related  to  the  use  of  active  or  passive 
sensors  on  the  aircraft  or  weapon  rails.  It  may  be  that  passive  systems  in  which  the  user 
scans  the  information  (e.g.  bar  codes)  rather  than  active  tags  that  radiate  information 
could  be  used.  Future  aircraft  are  being  designed  to  provide  information  about  their  on¬ 
board  systems  automatically  to  provide  maintenance  information.  Information  about 
configuration  could  also  be  included. 
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6.7  INTERFACING  WITH  EXISTING  SYSTEMS 

Flightline  maintainers  do  not  want  or  need  another  stand  alone  system.  SSLC2 
requirements  are  being  written  so  that  the  functions  can  be  incorporated  into  another 
system  such  as  Enhanced  Maintenance  Operation  Center  (EMOC)  or  Expeditionary 
Combat  Support  Systems  (ECSS).  It  will  be  important  for  SSEC2  to  interface  with  other 
systems  including  Core  Automated  Maintenance  System  (CAMS),  and  scheduling  tools. 
Eor  example,  the  interface  with  CAMS  would  save  a  great  deal  of  time  in  closing  out  a 
job.  When  the  user  selects  the  completion  flag  on  the  last  sub  task  of  the  maintenance 
task,  a  message  could  be  sent  to  CAMS  to  close  that  job  control  number.  Currently  they 
must  go  back  into  the  hangar,  log  onto  CAMS  and  close  out  the  job. 

Users  from  the  held  demonstration  at  Toledo  ANG  and  the  data  collection  trip  at 
Seymour  Johnson  Air  Eorce  Base  suggested  that  SSEC2  integrate  with  the  Tool 
Accountability  Systems  (TAS).  TAS  is  used  to  track  tools,  test  equipment  and  vehicles. 
TAS  requires  users  to  scan  in  their  personal  identiher  and  then  scan  in  the  tool  tag  and 
aircraft.  These  systems  could  be  linked  so  that  smart  tags  for  personnel  could  link  with 
the  TAS  tags  so  that  when  a  person  selects  an  item  the  system  would  immediately  link 
the  item  to  the  person  to  connect  the  two  identihers.  If  an  item  was  separated  from  the 
individual,  SSEC2  could  quickly  be  queried  to  identify  the  location  of  the  item.  This 
would  eliminate  the  need  for  scanning  and  personnel  would  not  have  to  hang  the  item 
identiher  on  their  key  chain  with  their  personal  identiher. 

The  SSEC2  research  focused  on  what  users  would  see  given  available  data.  It  will  be 
critical  to  determine  what  data  will  be  available  from  existing  systems  and  what  data  will 
require  manual  input. 

6.8  SAFETY  DRIVEN  TASKS 

There  are  some  aircraft  maintenance  tasks  that  are  driven  by  safety.  For  example, 
there  may  be  Time  Change  Technical  Orders  (TCTO)  or  Time  Change  Items  (TCI)  which 
require  specihc  maintenance  on  one  or  ah  aircraft.  These  orders  should  be  included  in 
SSEC2.  Originally  it  was  considered  to  put  this  data  in  the  aircraft  details  view.  However 
it  may  also  need  to  be  considered  for  a  maintenance  view.  This  information  and  how  it  is 
presented  should  be  considered  in  future  spirals. 
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6.9  FORM  2407  ROUTING 

The  users  identified  the  need  to  eleetronieally  eomplete,  submit,  route,  and  approve 
the  Form  2407  (change  of  schedule).  SSLC2  should  provide  the  ability  to  submit  an 
electronic  version  of  the  Form  2407  for  changes  made  to  the  flying  schedule  related  to 
takeoff  time  delays  as  well  as  aircraft  swaps.  Once  the  electronic  version  of  the  form  is 
submitted,  the  user’s  display  is  updated  with  the  changes,  but  will  signify  that  the  change 
is  not  yet  approved.  The  electronic  form  is  sent  automatically  to  all  associated  parties  for 
approvaFacknowledgement.  Once  all  parties  that  are  required  to  approve  the  change 
have  approved  it,  the  schedule  will  change  to  signify  that  the  changes  have  been 
accepted.  This  functionality  was  considered  for  Spiral  Three  but  not  fully  implemented 
based  on  time  and  cost  constraints.  The  2407  Form  was  not  routed  and  completed,  but 
Spiral  Three  did  provide  the  logic  for  submitting  requests  for  swap  approval  and 
displaying  the  schedule  change  as  soon  as  the  swap  was  approved  electronically. 

6.10  VISUALIZING  AND  CLEARING  CONFLICTS 

SSLC2  allowed  users  to  visualize  conflicts  on  the  maintenance  view,  personnel  view, 
and  equipment  view.  With  the  concepts  of  zoomable  user  interfaces  it  should  be  easy  for 
users  to  quickly  see  conflicts.  However,  it  is  possible  that  they  may  not  see  the  conflict 
depending  on  where  they  are  in  a  view.  For  example,  they  may  be  zoomed  into  a  specific 
aircraft  maintenance  task.  There  may  be  a  need  for  additional  ways  to  notify  the  user  of 
conflicts;  however,  this  need  requires  confirmation  once  zooming  is  available.  Alerting  is 
one  possibility.  Another  is  to  create  a  high  level  tab  that  summarizes  conflict,  or  creating 
a  tab  in  the  details  view.  Once  a  conflict  is  noticed,  the  user  should  be  able  to  deal  with 
the  conflict  at  the  point  in  the  interface  where  it  is  noticed.  Currently  they  have  to 
navigate  to  the  task  and  use  edit  subtask. 

6.11  FUTURE  RESEARCH 
6.11.1  Collaboration 

Complex  systems  such  as  fiightline  maintenance  require  significant  collaboration. 
There  is  considerable  need  for  research  related  to  collaboration  in  general,  and  continued 
need  for  research  related  to  collaboration  specifically  for  flightline  maintenance.  For 
example,  how  would  the  use  of  SSLC2  concepts  change  the  way  users  communicate  with 
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one  another.  Do  users  feel  that  the  system  provides  them  with  better  awareness  such  that 
large  amounts  of  radio  communication  are  not  needed?  How  might  Voice  Over  Internet 
Protocol  (VoIP)  be  used  to  reduce  radio  communications  and  allow  more  flightline 
personnel  to  communicate?  Currently  not  all  personnel  carry  radios. 

Collaboration  also  occurs  during  weekly  meetings,  the  ability  to  automatically  create 
performance  reports  based  on  flightline  metrics  may  be  beneficial  to  collaboration 
allowing  everyone  to  understand  what  issues  exist. 

6.11.2  Measuring  Decision  Making  Performance 

It  is  the  ultimate  goal  of  SSLC2  and  other  decision  support  systems  to  improve 

decision  making.  While  conducting  this  research  there  were  plans  to  try  and  measure 
decision  performance.  However,  this  focus  was  redirected  based  on  cost  and  time 
constraints.  While  there  is  a  large  amount  of  published  research  in  the  area  of  decision 
making,  it  is  actually  very  difficult  to  measure  human  decision  making  performance.  In 
many  cases  the  measures  are  either  subjective  (questionnaire  or  user  perceptions  of  their 
performance)  or  inferred  based  on  their  task  performance.  One  problem  is  that  people  can 
make  a  good  decision  that  results  in  a  bad  outcome,  and  they  can  make  a  bad  decision 
that  results  in  a  good  outcome.  To  ensure  that  decision  support  systems  are  making  a 
difference,  there  is  a  need  for  research  on  objectively  measuring  complex  decision 
making  in  field  environments. 

6.11.3  Automation 

Systems  such  as  SSLC2  should  consider  functions  and  tasks  that  can  be  automated. 
During  this  research,  alerts  and  messages  were  automatically  generated  based  on  certain 
tasks.  For  example,  if  the  Expeditor  assigned  a  resource  to  a  task  and  the  resource  was 
not  owned  by  the  Expeditor,  a  message  was  automatically  generated  requesting  the 
resource.  Other  tasks  could  also  be  automated.  Eor  example,  if  a  conflict  exists  among 
resources,  the  system  should  attempt  to  deal  with  the  conflict  by  reassigning.  This  is 
feasible  because  some  conflicts  arise  between  high  density/low  demand  resources.  Eor 
low  density,  high  demand  resources  the  system  could  recommend  changes  to  fix  conflicts 
by  summarizing  the  information  to  the  user  to  allow  them  to  make  the  final  choice.  Eor 
example,  the  system  could  determine  which  aircraft  has  the  next  sortie  and  help  the  user 
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choose  how  to  reassign  the  resouree.  Another  example  is  if  a  resource  ehanges  status 
(e.g.  resouree  is  broken)  the  system  automatieally  reassigns  another  resouree. 

6.1 1 .4  Smart  Sensor  Development 

As  diseussed  in  reeommendations  above  there  is  still  a  need  for  development  of 
sensors  that  ean  sense  information  other  than  loeation.  Also,  sensors  used  for  existing 
systems  eould  be  eonsidered.  For  example,  drivers  are  told  when  their  gas  is  low  and  how 
many  miles  until  empty.  This  teehnology  eould  be  ineorporated  into  equipment  that 
requires  fuel.  Battery  sensing  indieators  eould  also  be  used.  Researeh  into  the  use  of 
sensors  and  their  eonstraints  ean  be  direetly  applied  to  other  eomplex  systems. 


7  Conclusions 

The  goal  of  SSLC2  Spiral  Three  was  to  investigate  ways  to  support  logistie  deeision 
makers  in  eomplex  flightline  operations  through  the  fusion  of  existing  flightline 
information  with  smart  sensor  information.  SSLC2  provides  support  to  Expeditors  and 
ProSupers  on  the  flightline.  Spiral  Three  researeh  foeused  on  the  fusion  of  information 
and  presentation  of  the  information  through  interaetive  visualizations  to  support  situation 
awareness,  deeision  making  and  eollaboration  when  Expeditors  are  faeed  with  multiple 
eomplex  tasks  and  deeisions.  The  following  five  researeh  objeetives  were  met: 


Objeetive  1: 
Objeetive  2: 

Objeetive  3: 
Objeetive  4: 

Objeetive  5: 


Develop  smart  sensors  for  logisties  operations. 

Develop  a  method  for  determining  how  data  should  be  fused  for 
presentation  that  foeuses  on  supporting  deeisions  and  eollaboration. 
Obtain  user  opinions  of  deployed  sensors  in  the  field. 

Develop  a  simulation  test  bed  and  evaluate  user  feedbaek  on  integrated 
data  fusion  deeision  support  eoneepts. 

Develop  reeommendations  for  transition  to  existing  systems 


The  presentation  of  information  from  sensors  integrated  with  flightline  information 
and  an  understanding  of  how  users  perform  their  work  was  very  sueeessful  as 
demonstrated  in  both  Spiral  One  [8,10]  and  Spiral  Three.  Elsers  were  asked  to  rate  the 
effeetiveness  of  SSEC2  for  supporting  situation  awareness,  deeision  making, 
eollaboration,  the  effeetiveness  of  information  integration,  visualizations,  resouree 
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visibility  and  resource  allocation.  The  questionnaire  results,  comfort  ratings,  and  timing 
results  for  situation  awareness  probe  questions  indicate  that  overall  subjects  found  the 
SSLC2  system  to  be  effective  and  useful.  The  user’s  primary  suggestions  for 
improvement  were  related  to  functions  that  are  requirements  but  that  were  not 
demonstrated  in  the  simulation  due  to  time  and  cost  constraints  such  as  zooming,  sorting, 
and  filtering.  Users  did  not  indicate  that  any  information  was  missing.  Users  commented 
that  collaboration  through  messaging  would  require  auditory  tones  so  that  they  would 
know  when  a  message  was  received.  The  lack  of  auditory  tones  led  to  ratings  of 
“somewhat  effective”  under  collaboration.  However,  this  system  also  enhances 
collaboration  by  letting  different  personnel  and  workshops  view  shared  information  on 
resources  as  well  as  maintenance  tasks  and  assignments.  Users  felt  that  the  ability  for 
everyone  to  see  and  share  the  same  information  was  very  useful.  A  ProSuper  indicated 
that  there  would  be  considerable  time  savings  for  several  decisions  using  SSLC2 
including  the  decision  as  to  whether  an  aircraft  will  meet  a  sortie.  He  indicated  it  takes 
approximately  fifteen  (15)  minutes  to  collect  all  the  information  from  various  sources  to 
make  this  decision,  but  that  with  SSLC2  the  decision  could  be  made  in  seconds.  He  also 
indicated  there  would  be  at  least  a  60%  savings  in  the  time  it  takes  to  assign/reassign  and 
find  all  resources  needed  to  fix  aircraft. 

During  Spiral  Three,  active  tags  were  developed  that  provide  two  levels  of  status 
information,  although  not  demonstrated  due  to  time  and  cost  constraints,  active  tags  were 
simulated  in  the  test  bed  to  illustrate  how  status  information,  in  addition  to  location 
information,  could  improve  resource  allocation  and  visibility.  Additional  information 
that  can  be  sensed  was  also  described  in  Section  6  Recommendations. 

The  simulation  test  bed  approach  was  very  successful,  allowing  users  to  interact  with 
SSLC2  to  understand  the  design  concepts.  The  test  bed  is  a  functioning  hardware  and 
software  system  that  can  now  be  used  for  further  research. 

During  the  design  process  we  created  a  rating  technique  to  allow  SMEs  to  rate  the 
importance  of  information  across  decisions  and  then  performed  a  cluster  analysis.  The 
cluster  analysis  was  very  helpful  for  developing  constraints  and  ideas  for  integrating  large 
amounts  of  data  for  arrangement  in  the  interface. 
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7.1  CONTRIBUTIONS  OF  THE  RESEARCH 

There  are  many  eomplex  systems  that  ean  benefit  from  the  use  of  smart  sensors 
ineluding  logistics  for  air  and  space,  health  care,  disaster  management,  homeland 
security,  manufacturing,  and  industrial  processing  to  name  a  few.  The  SSLC2  program 
demonstrated  that  commercial  off-the-shelf  (COTS)  applications  without  thought  to 
integration  with  existing  systems  and  information  is  not  ideal  and  can  potentially  cause 
more  problems  or  even  rejection  of  the  technology.  Integration  of  information  is  key  for 
the  successful  use  of  sensing  technologies.  This  research  made  the  following 
contributions: 

•  The  research  illustrated  how  cluster  analysis  techniques  can  be  used  to  support 
visualization  design  by  helping  to  map  information  to  actual  decisions.  Since 
visualization  design  is  somewhat  of  an  art  form,  it  is  often  difficult  to  move 
from  cognitive  work  analysis  to  actual  design  [9]. 

•  The  research  identified  sensor  information  in  addition  to  location  that  would 
be  useful  for  systems  that  use  equipment  and  personnel  resources  to  perform 
work. 

•  The  research  illustrated  the  appropriateness  of  a  user-centered  approach  to 
design. 

•  The  research  culminated  in  recommendations  and  future  research  concepts 
that  can  be  applied  to  other  complex  systems. 

•  The  research  developed  designs  for  interactive  visualizations  that  can  be 
applied  to  other  domains. 

•  The  research  documented  in-depth  details  related  to  the  work  of  Expeditors 
and  ProSupers  including  decisions  they  make,  information  they  use,  and 
problems  they  face. 
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Acronyms 


Acronym 

Definition 

ACS 

Agile  Combat  Support 

ACWA 

Applied  Cognitive  Work  Analysis 

AFFOR 

Air  Poree  Porees 

AFRL/RHAL 

Air  Poree  Research  Laboratory  Logistics  Readiness  Branch 

AFSC 

Air  Poree  Specialty  Code 

AFSPC 

Air  Poree  Space  Command 

AGE 

Aerospace  Ground  Equipment 

AIT 

Automatic  Information  Technologies 

ALS 

Autonomic  Logistics  System 

ANG 

Air  National  Guard 

AOC 

Air  And  Space  Operations  Center 

API 

Application  Programming  Interface 

ASD 

Average  Sortie  Duration 

ATD 

Advanced  Technology  Demonstration 

ATO 

Air  Tasking  Order 

AV 

Avionics  Shop 

AWP/AWM 

Awaiting  Parts/ Awaiting  Materials 

BO 

Business  Objects 

CA/CRL 

Custodian  Authorization/Custody  Receipt  Listing 

CAMS 

Core  Automated  Maintenance  System 

CANN 

Cannibalized 

CC 

Commander 

COL 

Collaboration 

COTS 

Commercial  Off-The-Shelf 

CR 

Crew  Ready 

CS 

Crew  Show 

CTS 

Collaborative  Technologies  and  Systems 

CWA 

Cognitive  Work  Analysis 

DLL 

Dynamic  Linked  Library 

DS 

Decision  Support 

DSRC 

Dedicated  Short  Range  Communication 

E/E 

Electronic  And  Environmental 

ECSS 

Expeditionary  Combat  Support  Systems 

EE 

Electrical  Engineering 

EMOC 

Enhanced  Maintenance  Operations  Center 

EMOC 

Enhanced  Maintenance  Operation  Center 

ENG 

Engine  Shop 

ES 

Engine  Start 

ETIC 

Estimated  Time  Of  Completion 

EXP 

Experimenter 

PMC 

Pully  Mission  Capable 

POD 

Poreign  Object  Damage 
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Acronym 

Definition 

FSE 

Plying  Schedule  Effeetiveness 

GEO 

Geographie 

GPS 

Global  Positioning  System 

HTTP 

Hypertext  Transport  Protoeol 

IDE 

Integrated  Development  Environment 

IE 

Internet  Explorer 

IMIS 

Integrated  Maintenanee  Information  Systems 

INE 

Integration  Of  Information 

IRA 

Interfaee  Requirements  Agreements 

ITV 

In-  Transit  Visibility 

J2EE 

Java2  Platform  Enterprise  Edition 

JDBC 

Java  Database  Conneetivity 

JDK 

Java  Development  Kit 

JNDI 

Java  Naming  And  Directory  Interface 

JQS 

Job  Qualifieation  Standard 

JRE 

Java  Runtime  Environment 

JSPs 

Java  Server  Pages 

JVM 

Java  Virtual  Maehine 

KPP 

Key  Performanee  Parameters 

E 

Eanding 

EAN 

Eoeal  Area  Network 

ECOM 

Eogistics  Composite  Model 

ED/HD 

Eow  Density/High  Demand 

EED 

East  Plight  Date 

EG 

Eogisties  Group 

EOCIS 

Eibrary  of  Congress  Information  System 

MAJCOM 

Major  Command 

MC 

Mission  Capable 

MC 

Mission  Capable 

MDS 

Mission  Design  Series 

MES 

Mission  Essential  Subsystem 

MESE 

Mission  Essential  Subsystem  Eist 

MES 

Main  Eoeation  Sensor 

MOC 

Maintenanee  Operations  Center 

MX 

Maintenanee 

NIC 

Network  Interfaee  Card 

NMC 

Non-Mission  Capable 

NMC 

Non-Mission  Capable 

OG 

Operations  Group 

OTI 

One  Time  Inspeetion 

PEX 

Patriot  Excalibur 

PET 

Pilot  Plight  Training 

PMC 

Partially  Mission  Capable 

ProSupers 

Produetion  Supervisors 
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Acronym 

Definition 

PS 

Prosuper 

RA 

Resource  Allocation 

RF 

Radio  Frequency 

RFID/RTLS 

Radio  Frequency  Identification  /  Real  Time  Location  Systems 

RLA 

Resource  Location  Agent 

RV 

Resource  Visibility 

SA 

Situation  Awareness 

SID 

Oracle  System  ID 

SMC 

Missile  Systems  Center 

SMEs 

Subject  Matter  Experts 

SPAM 

Situation  Present  Assessment  Method 

SPARE 

Spare 

SSLC2 

Smart  Systems  For  Logistics  Command  And  Control 

STABOPS 

Stability  Operations 

TA/AS 

Table  of  Allowance/Allowance  Standard 

TAS 

Tool  Accountability  Systems 

TCI 

Time  Change  Item 

TCI 

Time  Change  Items 

TCP/IP 

Transmission  Control  Protocol/Intemet  Protocol 

TCTO 

Time  Compliance  Technical  Orders 

TCTO 

Time  Change  Technical  Orders 

TNMCM 

Total  Not  Mission  Capable  Rate  For  Maintenance 

TO 

Take-Off 

TOs 

Technical  Orders 

TX 

Taxi 

UMD 

Unit  Management  Document 

UML 

Unified  Modeling  Language 

URI 

Uniform  Resource  Identifier 

VAL 

Vehicle  Authorization  Listing 

VIS 

Visualizations 

VoIP 

Voice  Over  Internet  Protocol 

VSLRC 

Virtual  Space  Logistics  Readiness  Center 

WCSS 

Work  Centered  Support  Systems 

XC 

Experimental  Cargo 

ZUI 

Zoomable  User  Interface 
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Introduction 

Fusing  information  from  sensors  with  flightline  data  to  provide  shared  understanding 
among  personnel,  to  improve  deeision  making,  and  to  impact  logistic  operations  requires 
presenting  the  information  in  such  a  way  that  the  user  can  understand  the  information  and 
act  on  it.  Development  of  intuitive  interactive  visualizations  requires  a  systematic 
approach  that  includes  a  focus  on  the  user,  understanding  what  information  is  required  to 
make  decisions  and  to  enhance  effectiveness.  The  visualizations  must  integrate 
information,  providing  the  right  information  at  the  right  time.  While  there  are  general 
guidelines  when  developing  visualizations,  there  are  no  specific  rules  that  are  guaranteed 
to  provide  useful  intuitive  visualizations.  The  most  successful  visualizations  require  a 
user-centered  approach  which  includes  a  requirements  gathering  stage  to  develop  a 
thorough  understanding  of  the  information  users  need,  how  they  perform  their  work  and 
the  decisions  they  make.  This  requires  performing  up  front  cognitive  work  analysis  to 
understand  what  the  user  needs  and  transforming  the  data  gathered  into  actual  design  of 
user-interfaces  or  visualizations. 

ProSupers  and  Expeditors  make  many  decisions  requiring  significant  collaboration 
among  personnel  and  organizations.  Multiple  users  need  access  to  shared  information  and 
it  must  be  provided  beyond  one  day.  The  user-centered  approach  included  gathering 
additional  information,  beyond  Spiral  One,  related  to  decisions  made  and  information 
needed  for  Expeditors  and  ProSupers  by  reviewing  a  variety  of  fiightline  documents  from 
multiple  El.S.  bases,  training  manuals,  and  working  with  SMEs.  SMEs  included  personnel 
from  AERL  Eogistics  Research  Branch  as  well  as  discussion  with  personnel  working  a 
real  fiightline.  Erom  this  data  specific  decisions  were  gathered.  The  decisions  were 
presented  to  SMEs  who  then  narrowed  down  the  decisions  to  thirty-one  which  can  be 
found  in  Section  3.3  of  the  Einal  Report.  In  addition  to  establishing  the  decisions, 
information  or  data  elements  that  are  used  on  the  fiightline  were  complied.  A  list  of  157 
different  data  elements  was  created. 

Eor  input  to  the  design  of  the  visualizations  it  is  important  to  understand  how  the  data 
elements  map  to  the  decisions.  Not  all  information  is  needed  for  each  type  of  decision. 
The  goal  was  to  provide  users  with  easy  access  to  data  based  on  decisions  they  make. 
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In  order  to  perform  a  mapping  of  what  information  is  needed  for  different  decisions 
and  to  determine  the  importance  of  information  we  decided  to  use  a  cluster  analysis 
technique.  A  data  matrix  was  created  with  the  31  decisions  in  columns  and  the  157  data 
elements  in  rows.  Three  SMEs  were  then  asked  to  read  each  decision  and  rate  the 
importance  of  the  157  data  elements  to  that  decision. 

The  ratings  were  used  as  input  for  the  cluster  analysis.  A  hierarchical  cluster  analysis 
using  Ward’s  method  was  conducted  within  the  JMP  Professional  statistics  package 
(Version  5. 0.1. 2).  The  ratings  were  first  averaged  across  the  three  SMEs.  The  cluster 
analysis  was  conducted  across  all  decisions  to  be  made.  The  analysis  resulted  in  28 
clusters  of  the  157  data  items. 

This  Appendix  provides  a  summary  of  the  cluster  results.  The  table  includes  a 
column  which  provides  the  average  rating  of  subjects  responses  across  all  31  decisions. 
The  data  items  were  also  examined  by  the  average  rating  for  the  question  of  most 
“important  data  to  be  visible  most  of  the  time”,  which  is  also  included  in  Table  2. 

Data  Elements 

This  section  details  the  results  of  the  cluster  analysis  and  the  mapping  of  each  data 
element.  Table  1  provides  a  key  to  use  for  linkage  to  the  location  of  the  view  or  window 
of  the  SSEC2  interface.  The  details  screen  is  always  displayed  with  the  specific  view. 
Eor  example,  P  and  P-D  will  be  visible  at  the  same  time. 


Table  1.  User  Interface  View  Keys 


Key 

View 

Key 

View 

FS 

Flying  Schedule 

F-D 

Facilities  Detail 

FS-D 

Flying  Schedule  Detail 

MO 

Maintenance  Overview 

EQ 

Equipment 

MO-D 

Maintenance  Overview  Detail 

EQ-D 

Equipment  Detail 

MS 

Maintenance  Subtask 

P 

Personnel 

MS-D 

Maintenance  Subtask  Detail 

P-D 

Personnel  Detail 

Geo 

Geographic  view 

F 

Facilities 

Com 

Communication  window 

Table  2  lists  all  28  clusters,  the  data  elements  included  in  that  cluster,  the  average 
rating  by  SMEs  across  all  decisions  per  data  element,  the  average  rating  by  the  SMEs  on 
whether  the  data  element  should  be  visible  most  of  the  time,  and  the  location  of  that  data 
element  on  the  user  interface.  The  location  for  information  items  that  were  not  included 
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in  Spiral  Three  are  left  blank,  sinee  some  items  were  rated  as  not  at  all  important.  (Less 
than  2) 


Table  2.  Data  Element  Results 


Cluster 

Data  Element 

Average 
rating 
across  all 
decisions 

Average 

rating 

“visible 

info 

most  of 
the 
time” 

Location  on  User 
Interface 

1 

Tail  number  (element  aircraft) 

4.38 

5.00 

FS,  FS-D,  MO,  MO-D, 
MS,  MS-D,  F,  Geo 

Status  (PMC,  NMC,  FMC,  spare, 
CANN) 

4.06 

5.00 

FS,  FS-D,  MO,  MO-D, 
MS,  MS-D 

Reason  for  status 

4.23 

5.00 

FS-D,  MO,  MO-D,  MS, 
MS-D 

Schedule  for  day  (Current  time  and 
date,  all  aircraft.  Status,  brief  time, 
take  off  time,  land  time  duration  of 
flight,  load,  remarks,  turn,  weather. 
Phase?) 

4.19 

5.00 

FS 

2 

Priority  of  task 

3.95 

3.00 

FS-D 

ETIC 

4.15 

5.00 

FS,  FS-D 

Tasks  required  to  solve  problem 

4.23 

4.33 

MS 

Current  Discrepancies 

4.48 

4.33 

MS 

Nature  of  problem. 

4.48 

5.00 

MO,  MO-D,  MS,  MS-D 

3 

Corrective  actions  (maintenance) 

3.49 

3.67 

MS 

Discrepancies  awaiting  parts 

3.58 

2.67 

MS  (Notes) 

Scheduled  Maintenance 

3.57 

4.67 

MO,  MS 

Safety/Danger  Considerations 
(power/hydraulic  applications, 
stress  panels  removed,  aircraft  on 
jacks,  weight  and  balance,  etc.) 

3.56 

3.00 

MS  (Notes) 

Discrepancies  awaiting 
maintenance 

3.38 

3.33 

MO 

4 

Personnel  current  a/c  working  or 
task 

3.47 

2.67 

P,  P-D,  MS,  MS-D 

Personnel  Availability 

3.34 

2.67 

P,  P-D,  MS  (Edit 
subtask) 

Supplies  on  order  and  expected 
delivery  date 

2.10 

1.33 

Ml  CAPS 

3.53 

3.67 

Status  of  backorder  parts 

3.03 

1.00 

5 

Availability  of  critical  equipment 

2.90 

2.00 

EQ,  EQ-D 
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Cluster 

Data  Element 

Average 
rating 
across  all 
decisions 

Average 

rating 

“visible 

info 

most  of 
the 
time” 

Location  on  User 
Interface 

List  of  CANNED  items 

2.61 

2.33 

Stage  of  CANN  jet 

2.38 

2.00 

6 

Schedule  for  the  week 

2.91 

2.33 

FS 

Mission  Requirements 

2.73 

2.33 

Weather  forecast 

2.70 

3.67 

Next  to  communication 
always  visible. 

7 

Rules  for  exceptional  releases 

2.20 

1.00 

Ground  Time  Requirements 

2.19 

1.67 

8 

Aircraft  load 

2.19 

3.00 

FS 

Type  of  aircraft  (Model) 

2.49 

4.00 

FS-D 

Aircraft  with  pacer  motors  identified. 

2.05 

3.67 

FS-D 

A/C  Projected  to  be  USM 

2.47 

4.33 

9 

Load 

2.27 

2.67 

FS 

Type  of  sortie  on  schedules  (night, 
XC,  training) 

2.48 

3.10 

FS-D 

MESL  (for  mission,  basic  and  full) 

2.65 

1.67 

Current  Configuration 

2.89 

4.33 

FS,  FS-D 

Location  on  flight  line 

2.84 

4.33 

FS,  FS-D,  GEO 

10 

take-off  time 

3.31 

5.00 

FS 

Priority  of  sorties 

3.32 

5.00 

11 

Schedule  changes 

3.27 

4.33 

FS 

Lines  required  for  filling 

3.10 

4.33 

FS 

Service  item  status,  requirement  for 
next  flight 

3.10 

3.67 

12 

Land  time 

2.71 

5.00 

FS 

Turn 

2.55 

4.67 

FS 

Line  number  (details  type  of  sortie) 

2.55 

4.67 

FS,FS-D 

A/C  Projected  to  be  Spares 

2.71 

4.33 

FS 

A/C  Projected  to  be  Prime  Flyers 

2.71 

4.33 

FS 

13 

schedule  for  the  month 

1.88 

1.67 

FS 

Rules  for  Status  Reporting 

1.95 

1.00 

Rules  for  deviation  reporting 

1.62 

1.00 

14 

Past  week  maintenance  history 

2.33 

1.33 

Scheduling  guidelines 

2.18 

1.00 
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Cluster 

Data  Element 

Average 
rating 
across  all 
decisions 

Average 

rating 

“visible 

info 

most  of 
the 
time” 

Location  on  User 
Interface 

Top  Supply  Drivers/Parts  that  have 
problems 

2.42 

2.33 

Work  Unit  Code 

2.42 

3.33 

MS-D 

Supplies  in  stock,  Inventory  levels 

3.03 

1.00 

Weight  and  balance  information 

1.87 

1.00 

15 

Crew  chief  currently  assigned 

2.47 

3.00 

FS-D,  MO-D,  MS-D 

Personnel  role  (crew  chief, 
specialist,  etc) 

2.63 

3.00 

P,  PD,  FS,  MS-D 

Personnel  qualifications, 
certifications 

2.74 

2.33 

P,  PD,  MS-D 

Qualification  Levels  (3-level,  5- 
level,  7-level) 

2.81 

1.67 

P,  PD,  MS-D 

16 

Work  schedules  (shift) 

2.03 

3.33 

P 

Shifts  Scheduled 

2.39 

3.00 

P 

Personnel  Contact  info  (name, 
radio  call,  or  other) 

2.11 

1.00 

P-D 

Personnel  Current  location 

2.22 

1.00 

P-D,  GEO 

AFSC 

2.23 

2.00 

P-D 

17 

Scheduled  Maintenance 
Requirements  (Generic  high  level 
used  to  create  yearly  forecast  - 
example:  average  1  A/C  out  per 
month  for  depot ) 

2.01 

3.33 

Maintenance  Training 

Requirements 

2.20 

4.33 

TCTO  (Time  Compliance  Technical 
Orders)  -  Safety  Issues  -  Contains 
A/C  Tail  Number,  Date 

2.77 

3.33 

FS-D? 

OTI  (One  Time  Inspection)  Tail 
Number,  Date 

2.76 

3.33 

FS-D? 

TCI  (Time  Change  Item) 

2.69 

3.33 

FS-D? 

18 

Last  phase  inspection 

1.55 

2.00 

FS-D 

Next  phase  inspection 

1.56 

2.00 

FS-D 

Phase  Schedule 

1.72 

2.00 

FS-D 

Engine  Hours 

1.63 

3.00 

FS-D 

Airframe  flight  hours 

1.77 

2.33 

FS-D 
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Cluster 

Data  Element 

Average 
rating 
across  all 
decisions 

Average 

rating 

“visible 

info 

most  of 
the 
time” 

Location  on  User 
Interface 

Days/time  since  last  flown  (hangar 
queen) 

2.14 

3.00 

FS-D 

Last  flight  date  (LFD)  When  aircraft 
flew  last 

1.78 

3.00 

FS-D 

19 

Facilities  available 

1.90 

1.67 

F 

Facilities  schedule 

1.69 

1.67 

F 

Facilities  needed  for  maintenance 

1.83 

1.67 

MS,  MS-D 

20 

Equipment  Type 

1.66 

1.00 

EQ,  EQ-D 

Equip  performance  capability 

1.75 

1.00 

Equip  Current  status  (broke,  not 
broke) 

2.03 

1.00 

EQ,  EQ-D 

Equip  Availability 

2.05 

2.33 

EQ,  EQ-D 

Equip  Location 

1.90 

2.00 

EQ-D,  GEQ 

21 

crew  brief  time  for  aircraft 

1.68 

2.00 

FS 

Passengers 

1.78 

2.33 

FS-D 

Turn  Patterns 

1.67 

4.67 

FS 

Cross-County  Sorties 

1.57 

3.33 

FS-D 

Night-flying  Requirements 

1.62 

3.33 

Flybys/Special  Activities 

1.62 

3.33 

FS,  FS-D 

22 

Duration  of  flight 

1.45 

1.67 

FS 

Authorized  configurations 

1.74 

1.33 

Projected  Turn  Patterns  for  each 
O&M  Day 

1.84 

2.67 

Squadron  Requirements 

1.84 

1.67 

Landing  codes  (0  through  5) 

1.87 

2.33 

Diverted  aircraft 

1.95 

3.33 

23 

Deviations  (type,  Code) 

1.47 

1.67 

System  capability  codes  (0-8) 

1.42 

1.00 

Flight  crew  for  sortie 

1.24 

2.33 

FS-D 

Days  and  Swings  FOD  walk  times 

1.45 

2.33 

CQM 

Quick  Reference  list 

1.65 

2.33 

D-Model  Constraints 

1.55 

2.33 

Heavy  D-model  requirement  days 
identified  and  supported 

1.39 

2.00 

Spare  D-model  allocation 

1.49 

2.00 
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Cluster 

Data  Element 

Average 
rating 
across  all 
decisions 

Average 

rating 

“visible 

info 

most  of 
the 
time” 

Location  on  User 
Interface 

Heavy  pilot  flight  training  (PFT) 
Days 

1.24 

2.33 

Repeat  rate 

1.52 

1.67 

MS-D 

Recur  rate 

1.51 

1.67 

MS-D 

24 

Aircraft  call  sign 

1.23 

3.00 

FS-D 

Assigned  facilities  (base  &  unit) 

1.32 

1.00 

F 

Hours  to  Phase 

1.33 

1.33 

FS-D 

MSE  rate 

1.41 

1.00 

Operating  hours 

1.13 

1.67 

Depot  Schedule  (Tail  Number  and 
Target  Dates-War  time  has  different 
rules) 

1.18 

2.00 

Depot  inputs 

1.19 

3.00 

Non-Flying  Days 

1.31 

2.33 

FS 

25 

Leave  schedules 

1.40 

1.33 

P 

Training  schedules 

1.70 

2.33 

P 

Upcoming  personnel  Deployments 
or  TDY’s 

1.56 

1.67 

P 

26 

Percentage  of  time  a/c  in  each 
state  (FMC,  PMC,  NMC,  CANN, 
hangar  queen) 

1.87 

1.67 

CANN  rate 

1.48 

1.00 

8-/12  hour  fix  rate 

1.51 

2.00 

Average  deferred/delayed 
discrepancies  per  aircraft 

1.31 

1.00 

Ml  CAP  aircraft  part  rate 

1.44 

1.00 

Average  repair  cycle  days 

1.27 

1.00 

Phase  flow  (time  distribution 
interval) 

1.09 

1.33 

27 

Crew  chief  assigned  past  week,  or 
rest  of  this  week  or  month 

1.12 

1.00 

reported  fleet  average  maintenance 
status 

1.14 

1.00 

Over  time  for  the  week 

1.18 

1.00 

Authorized  manning 

1.12 

1.00 

Flying  hour  Allocation 

1.10 

1.00 
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Cluster 

Data  Element 

Average 
rating 
across  all 
decisions 

Average 

rating 

“visible 

info 

most  of 
the 
time” 

Location  on  User 
Interface 

Position  Numbers  from  Unit 
Management  Document  (UMD) 

0.96 

1.00 

Table  of  Allowance/Allowance 
Standard  (TA/AS) 

1.00 

1.00 

supply  use  rates 

1.03 

0.67 

Custodian  Authorization/Custody 
Receipt  Listing  (CA/CRL) 

1.02 

1.00 

Vehicle  Authorization  Listing  (VAL) 

1.03 

1.00 

Attrition  Calculations 

1.11 

1.00 

Primary  a/c  inventory  versus 
possessed  aircraft  rate 

1.09 

1.00 

Chargeable  deviation  rate 

1.22 

1.00 

Nonchargable  deviation  rate 

1.17 

1.00 

Who  owns  Equip  (this  line,  or 
another  line  or  squadron) 

1.31 

1.00 

Equip  recent  and  historical 
utilization  rate 

1.28 

1.00 

Deployment  Plans 

1.28 

1.00 

authorized  equipment 

1.08 

1.00 

Ground  Abort  rate 

1.25 

1.33 

Air  abort  rate 

1.23 

1.33 

Code  3  break  rate 

1.29 

1.33 

MAP  total  air  abort  rate  (home 
station  air  aborts  +J  diverts) 

1.05 

1.00 

Logistics  departure  reliability 

1.12 

1.00 

Functional  check  flight  release  rate 

1.14 

1.00 

Issue  effectiveness  rate 

1.12 

1.00 

Stockage  effectiveness  rate 

1.08 

1.00 

Bench  stock  effectiveness  rate 

1.10 

1.00 

28 

Programmed  UTE  versus  actual 

UTE 

1.25 

1.00 

Programmed  average  sortie 
duration  (ASD)  vs  actual  ASD 

1.18 

1.00 

Flying  hour  execution 

1.27 

1.00 

Flying  schedule  effectiveness  rate 
(FSE  rate) 

1.22 

1.00 
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